Update README.md

README.md

@@ -10,4 +10,136 @@ pinned: false
 license: apache-2.0
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Orthogonal Adaptation
+
+## 🔧 Dependencies and Installation
+
+- Python >= 3.9 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html))
+- Diffusers==0.19.3
+- XFormer (is recommend to save memory)
+
+## ⏬ Pretrained Model and Data Preparation
+
+### Pretrained Model Preparation
+
+We adopt the [ChilloutMix](https://civitai.com/models/6424/chilloutmix) fine-tuned model for generating human subjects.
+
+```bash
+git clone https://github.com/TencentARC/Mix-of-Show.git
+
+cd experiments/pretrained_models
+
+# Diffusers-version ChilloutMix
+git-lfs clone https://huggingface.co/windwhinny/chilloutmix.git
+```
+<!-- ### Data Preparation
+
+Note: Data selection and tagging are important in single-concept tuning. We strongly recommend checking the data processing in [sd-scripts](https://github.com/kohya-ss/sd-scripts). **In our ED-LoRA, we do not require any regularization dataset.** The detailed dataset preparation steps can refer to [Dataset.md](docs/Dataset.md). Our preprocessed data used in this repo is available at [Google Drive](https://drive.google.com/file/d/1O5oev8861N_KmKtqefb45l3SiSblbo5O/view?usp=sharing). -->
+
+## 🕹️ Single-Client Concept Tuning
+### Step 0: Data selection and Tagging for a single concept
+Data selection and tagging are crucial in single-concept tuning. We strongly recommend checking the data processing in [sd-scripts](https://github.com/kohya-ss/sd-scripts). **In our ED-LoRA, we do not require any regularization dataset.** 
+
+1. **Collect Images**: Gather 5-10 images of the concept you want to customize and place them inside a single folder located at `/single-concept/data/yourconceptname/image`. Ensure the images are consistent but also varied in appearance to prevent overfitting.
+
+2. **Create Captions**: Write captions for each image you collected. Save these captions as text files in the `/single-concept/data/yourconceptname/caption` directory.
+
+3. **Generate Masks**: To further improve the understanding of the concept, save masks of each image in the `/single-concept/data/yourconceptname/mask` directory. Use the `data_processing.ipynb` notebook for this step.
+
+4. **Create Data Configs**: In the `/single-concept/data_configs` directory, create a JSON file that summarizes the files you just created. The file name could be `yourconceptname.json`.
+
+
+### Step 1: Modify the Config at `/single-concept/train_configs`
+
+Before tuning, it is essential to specify the data paths and adjust certain hyperparameters in the corresponding config file. Below are some basic config settings to be modified:
+
+- **Concept List**: The data config that you just created should be referenced from 'concept_list'.
+- **Validation Prompt**: You might want to set a proper validation prompt in `single-concept/validation_prompts` to visualize the single-concept sampling.
+
+
+```yaml
+manual_seed: 1234 # this seed determines choice of columns from orthogonal basis (set differently for each concept)
+
+datasets:
+  train:
+    # Concept data config
+    concept_list: single-concept/data_configs/hina_amano.json
+    replace_mapping:
+      <TOK>: <hina1> <hina2> # concept new token
+  val_vis:
+    # Validation prompt for visualization during tuning
+    prompts: single-concept/validation_prompts/characters/test_girl.txt
+    replace_mapping:
+      <TOK>: <hina1> <hina2> # Concept new token
+
+models:
+  enable_edlora: true  # true means ED-LoRA, false means vallina LoRA
+  new_concept_token: <hina1>+<hina2> # Concept new token, use "+" to connect
+  initializer_token: <rand-0.013>+girl
+  # Init token, only need to revise the later one based on the semantic category of given concept
+
+val:
+  val_during_save: true # When saving checkpoint, visualize sample results.
+  compose_visualize: true # Compose all samples into a large grid figure for visualization
+```
+
+### Step 2: Start Tuning
+
+We tune each concept with 2 A100 GPU. Similar to LoRA, community user can enable gradient accumulation, xformer, gradient checkpoint for tuning on one GPU.
+
+```bash
+accelerate launch train_edlora.py -opt single-concept/0005_lebron_ortho.yml
+```
+The LoRA weights for the single concept are saved inside the `/experiments/single-concept` folder under your concept name folder
+
+### Step 3: Single-concept Sampling
+To sample an image from your trained weights from the last step, specify the model path in the sample config (located in `/single-concept/sample_configs`) and run the following command:
+
+```bash
+python test_edlora.py -opt single-concept/sample_configs/8101_EDLoRA_potter_Cmix_B4_Repeat500.yml
+```
+
+## 🕹️ Merging LoRAs
+
+### Step 1: Collect Concept Models
+
+Collect all concept models you want to extend the pretrained model and modify the config in `/multi-concept/merge_configs` accordingly.
+
+```yaml
+[
+    {
+        "lora_path": "experiments/0022_elsa_ortho/models/edlora_model-latest.pth",
+        "unet_alpha": 1.8,
+        "text_encoder_alpha": 1.8,
+        "concept_name": "<elsa1> <elsa2>"
+    },
+    {
+        "lora_path": "experiments/0023_moana_ortho/models/edlora_model-latest.pth",
+        "unet_alpha": 1.8,
+        "text_encoder_alpha": 1.8,
+        "concept_name": "<moana1> <moana2>"
+    }
+    ... # keep adding new concepts for extending the pretrained models
+]
+```
+
+### Step 2: Weight Fusion
+Specify which merge config you are using inside the `fuse.sh` file, and then run:
+
+```bash
+bash fuse.sh
+```
+The merged weights are now saved in the `/experiments/multi-concept` directory. This process is almost instant.
+
+### Step 3: Sample
+
+**Regionally controllable multi-concept sampling:**
+
+We utilize regionally controllable sampling from Mix-of-Show to enable multi-concept generation. Adding openpose conditioning greatly increases the reliability of generations.
+
+Define which fused model inside `/experiments/multi-concept` you are going to use, and specify the keypose condition in `/multi-concept/pose_data` if needed. Also, modify the context prompts and regional prompts. Then run:
+
+```bash
+bash regionally_sample.sh
+```
+The samples from the multi-concept generation will now be stored in the `/results` folder.