Hugging Face's logo

Spaces:
ujin-song
/
ortha
Sleeping

App Files Community
ujin-song commited on
Commit
b96bfe3
•
1 Parent(s): 26e2145

added Mirabel and Miguel weights

Browse files
Files changed (7) hide show
  1. experiments/single-concept/miguel/.DS_Store +0 -0
  2. experiments/single-concept/miguel/3644_miguel_ortho.yml +98 -0
  3. experiments/single-concept/miguel/models/edlora_model-latest.pth +3 -0
  4. experiments/single-concept/miguel/train_3644_miguel_ortho_20240616_042156.log +192 -0
  5. experiments/single-concept/mirabel/1058_mirabel_ortho.yml +98 -0
  6. experiments/single-concept/mirabel/models/edlora_model-latest.pth +3 -0
  7. experiments/single-concept/mirabel/train_1058_mirabel_ortho_20240616_051543.log +192 -0
experiments/single-concept/miguel/.DS_Store ADDED
Binary file (6.15 kB). View file
 
experiments/single-concept/miguel/3644_miguel_ortho.yml ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # GENERATE TIME: Sun Jun 16 04:21:56 2024
2
+ # CMD:
3
+ # train_edlora.py -opt single-concept/train_configs/3644_miguel_ortho.yml
4
+
5
+ name: 3644_miguel_ortho
6
+ manual_seed: 3644
7
+ mixed_precision: fp16
8
+ gradient_accumulation_steps: 1
9
+ datasets:
10
+ train:
11
+ name: LoraDataset
12
+ concept_list: single-concept/data_configs/miguel.json
13
+ use_caption: true
14
+ use_mask: true
15
+ instance_transform:
16
+ - type: HumanResizeCropFinalV3
17
+ size: 512
18
+ crop_p: 0.5
19
+ - type: ToTensor
20
+ - type: Normalize
21
+ mean:
22
+ - 0.5
23
+ std:
24
+ - 0.5
25
+ - type: ShuffleCaption
26
+ keep_token_num: 1
27
+ - type: EnhanceText
28
+ enhance_type: human
29
+ replace_mapping:
30
+ <TOK>: <miguel1> <miguel2>
31
+ batch_size_per_gpu: 2
32
+ dataset_enlarge_ratio: 500
33
+ val_vis:
34
+ name: PromptDataset
35
+ prompts: single-concept/validation_prompts/characters/test_boy_disney.txt
36
+ num_samples_per_prompt: 8
37
+ latent_size:
38
+ - 4
39
+ - 64
40
+ - 64
41
+ replace_mapping:
42
+ <TOK>: <miguel1> <miguel2>
43
+ batch_size_per_gpu: 4
44
+ models:
45
+ pretrained_path: nitrosocke/mo-di-diffusion
46
+ enable_edlora: true
47
+ finetune_cfg:
48
+ text_embedding:
49
+ enable_tuning: true
50
+ lr: 0.001
51
+ text_encoder:
52
+ enable_tuning: true
53
+ lora_cfg:
54
+ rank: 5
55
+ alpha: 1.0
56
+ where: CLIPAttention
57
+ lr: 1.0e-05
58
+ unet:
59
+ enable_tuning: true
60
+ lora_cfg:
61
+ rank: 5
62
+ alpha: 1.0
63
+ where: Attention
64
+ lr: 0.0001
65
+ new_concept_token: <miguel1>+<miguel2>
66
+ initializer_token: <rand-0.013>+boy
67
+ noise_offset: 0.01
68
+ attn_reg_weight: 0.01
69
+ reg_full_identity: false
70
+ use_mask_loss: true
71
+ gradient_checkpoint: false
72
+ enable_xformers: true
73
+ path:
74
+ pretrain_network: null
75
+ train:
76
+ optim_g:
77
+ type: AdamW
78
+ lr: 0.0
79
+ weight_decay: 0.01
80
+ betas:
81
+ - 0.9
82
+ - 0.999
83
+ unet_kv_drop_rate: 0
84
+ scheduler: linear
85
+ emb_norm_threshold: 0.55
86
+ val:
87
+ val_during_save: false
88
+ compose_visualize: false
89
+ alpha_list:
90
+ - 0
91
+ - 0.7
92
+ - 1.0
93
+ sample:
94
+ num_inference_steps: 50
95
+ guidance_scale: 7.5
96
+ logger:
97
+ print_freq: 10
98
+ save_checkpoint_freq: 10000.0
experiments/single-concept/miguel/models/edlora_model-latest.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:41fdde99ee841bac4752b918ee7f57aed2d6a87683d986d4f405dad114ca5c4a
3
+ size 35173046
experiments/single-concept/miguel/train_3644_miguel_ortho_20240616_042156.log ADDED
@@ -0,0 +1,192 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2024-06-16 04:21:56,051 INFO: Distributed environment: MULTI_GPU Backend: nccl
2
+ Num processes: 2
3
+ Process index: 0
4
+ Local process index: 0
5
+ Device: cuda:0
6
+
7
+ Mixed precision type: fp16
8
+
9
+ 2024-06-16 04:21:56,054 INFO:
10
+ name: 3644_miguel_ortho
11
+ manual_seed: 3644
12
+ mixed_precision: fp16
13
+ gradient_accumulation_steps: 1
14
+ datasets:[
15
+ train:[
16
+ name: LoraDataset
17
+ concept_list: single-concept/data_configs/miguel.json
18
+ use_caption: True
19
+ use_mask: True
20
+ instance_transform: [{'type': 'HumanResizeCropFinalV3', 'size': 512, 'crop_p': 0.5}, {'type': 'ToTensor'}, {'type': 'Normalize', 'mean': [0.5], 'std': [0.5]}, {'type': 'ShuffleCaption', 'keep_token_num': 1}, {'type': 'EnhanceText', 'enhance_type': 'human'}]
21
+ replace_mapping:[
22
+ <TOK>: <miguel1> <miguel2>
23
+ ]
24
+ batch_size_per_gpu: 2
25
+ dataset_enlarge_ratio: 500
26
+ ]
27
+ val_vis:[
28
+ name: PromptDataset
29
+ prompts: single-concept/validation_prompts/characters/test_boy_disney.txt
30
+ num_samples_per_prompt: 8
31
+ latent_size: [4, 64, 64]
32
+ replace_mapping:[
33
+ <TOK>: <miguel1> <miguel2>
34
+ ]
35
+ batch_size_per_gpu: 4
36
+ ]
37
+ ]
38
+ models:[
39
+ pretrained_path: nitrosocke/mo-di-diffusion
40
+ enable_edlora: True
41
+ finetune_cfg:[
42
+ text_embedding:[
43
+ enable_tuning: True
44
+ lr: 0.001
45
+ ]
46
+ text_encoder:[
47
+ enable_tuning: True
48
+ lora_cfg:[
49
+ rank: 5
50
+ alpha: 1.0
51
+ where: CLIPAttention
52
+ ]
53
+ lr: 1e-05
54
+ ]
55
+ unet:[
56
+ enable_tuning: True
57
+ lora_cfg:[
58
+ rank: 5
59
+ alpha: 1.0
60
+ where: Attention
61
+ ]
62
+ lr: 0.0001
63
+ ]
64
+ ]
65
+ new_concept_token: <miguel1>+<miguel2>
66
+ initializer_token: <rand-0.013>+boy
67
+ noise_offset: 0.01
68
+ attn_reg_weight: 0.01
69
+ reg_full_identity: False
70
+ use_mask_loss: True
71
+ gradient_checkpoint: False
72
+ enable_xformers: True
73
+ ]
74
+ path:[
75
+ pretrain_network: None
76
+ experiments_root: /home/ujinsong/workspace/ortha/experiments/3644_miguel_ortho
77
+ models: /home/ujinsong/workspace/ortha/experiments/3644_miguel_ortho/models
78
+ log: /home/ujinsong/workspace/ortha/experiments/3644_miguel_ortho
79
+ visualization: /home/ujinsong/workspace/ortha/experiments/3644_miguel_ortho/visualization
80
+ ]
81
+ train:[
82
+ optim_g:[
83
+ type: AdamW
84
+ lr: 0.0
85
+ weight_decay: 0.01
86
+ betas: [0.9, 0.999]
87
+ ]
88
+ unet_kv_drop_rate: 0
89
+ scheduler: linear
90
+ emb_norm_threshold: 0.55
91
+ ]
92
+ val:[
93
+ val_during_save: False
94
+ compose_visualize: False
95
+ alpha_list: [0, 0.7, 1.0]
96
+ sample:[
97
+ num_inference_steps: 50
98
+ guidance_scale: 7.5
99
+ ]
100
+ ]
101
+ logger:[
102
+ print_freq: 10
103
+ save_checkpoint_freq: 10000.0
104
+ ]
105
+ is_train: True
106
+
107
+ 2024-06-16 04:22:02,370 INFO: <miguel1> (49408-49423) is random initialized by: <rand-0.013>
108
+ 2024-06-16 04:22:02,879 INFO: <miguel2> (49424-49439) is random initialized by existing token (boy): 1876
109
+ 2024-06-16 04:22:02,885 INFO: optimizing embedding using lr: 0.001
110
+ 2024-06-16 04:22:03,076 INFO: optimizing text_encoder (48 LoRAs), using lr: 1e-05
111
+ 2024-06-16 04:22:03,354 INFO: optimizing unet (128 LoRAs), using lr: 0.0001
112
+ 2024-06-16 04:22:04,860 INFO: ***** Running training *****
113
+ 2024-06-16 04:22:04,860 INFO: Num examples = 3000
114
+ 2024-06-16 04:22:04,861 INFO: Instantaneous batch size per device = 2
115
+ 2024-06-16 04:22:04,861 INFO: Total train batch size (w. parallel, distributed & accumulation) = 4
116
+ 2024-06-16 04:22:04,861 INFO: Total optimization steps = 750.0
117
+ 2024-06-16 04:22:17,872 INFO: [3644_..][Iter: 10, lr:(9.867e-04,9.867e-06,9.867e-05,)] [eta: 0:14:34] loss: 1.2998e+00 Norm_mean: 3.7457e-01
118
+ 2024-06-16 04:22:28,967 INFO: [3644_..][Iter: 20, lr:(9.733e-04,9.733e-06,9.733e-05,)] [eta: 0:13:56] loss: 3.6238e-01 Norm_mean: 3.9041e-01
119
+ 2024-06-16 04:22:40,262 INFO: [3644_..][Iter: 30, lr:(9.600e-04,9.600e-06,9.600e-05,)] [eta: 0:13:41] loss: 1.6014e+00 Norm_mean: 4.0134e-01
120
+ 2024-06-16 04:22:51,643 INFO: [3644_..][Iter: 40, lr:(9.467e-04,9.467e-06,9.467e-05,)] [eta: 0:13:28] loss: 1.2411e+00 Norm_mean: 4.0942e-01
121
+ 2024-06-16 04:23:02,989 INFO: [3644_..][Iter: 50, lr:(9.333e-04,9.333e-06,9.333e-05,)] [eta: 0:13:16] loss: 8.9792e-01 Norm_mean: 4.1655e-01
122
+ 2024-06-16 04:23:14,193 INFO: [3644_..][Iter: 60, lr:(9.200e-04,9.200e-06,9.200e-05,)] [eta: 0:13:03] loss: 9.7455e-01 Norm_mean: 4.2226e-01
123
+ 2024-06-16 04:23:25,549 INFO: [3644_..][Iter: 70, lr:(9.067e-04,9.067e-06,9.067e-05,)] [eta: 0:12:51] loss: 3.1231e-01 Norm_mean: 4.2708e-01
124
+ 2024-06-16 04:23:37,018 INFO: [3644_..][Iter: 80, lr:(8.933e-04,8.933e-06,8.933e-05,)] [eta: 0:12:41] loss: 1.1681e+00 Norm_mean: 4.3169e-01
125
+ 2024-06-16 04:23:48,613 INFO: [3644_..][Iter: 90, lr:(8.800e-04,8.800e-06,8.800e-05,)] [eta: 0:12:31] loss: 4.6120e-01 Norm_mean: 4.3651e-01
126
+ 2024-06-16 04:24:00,295 INFO: [3644_..][Iter: 100, lr:(8.667e-04,8.667e-06,8.667e-05,)] [eta: 0:12:21] loss: 1.3633e-01 Norm_mean: 4.4076e-01
127
+ 2024-06-16 04:24:12,111 INFO: [3644_..][Iter: 110, lr:(8.533e-04,8.533e-06,8.533e-05,)] [eta: 0:12:12] loss: 3.6840e-01 Norm_mean: 4.4429e-01
128
+ 2024-06-16 04:24:23,982 INFO: [3644_..][Iter: 120, lr:(8.400e-04,8.400e-06,8.400e-05,)] [eta: 0:12:03] loss: 8.9865e-01 Norm_mean: 4.4807e-01
129
+ 2024-06-16 04:24:35,588 INFO: [3644_..][Iter: 130, lr:(8.267e-04,8.267e-06,8.267e-05,)] [eta: 0:11:52] loss: 1.0239e+00 Norm_mean: 4.5220e-01
130
+ 2024-06-16 04:24:47,400 INFO: [3644_..][Iter: 140, lr:(8.133e-04,8.133e-06,8.133e-05,)] [eta: 0:11:42] loss: 4.6487e-01 Norm_mean: 4.5623e-01
131
+ 2024-06-16 04:24:59,290 INFO: [3644_..][Iter: 150, lr:(8.000e-04,8.000e-06,8.000e-05,)] [eta: 0:11:31] loss: 1.0952e-01 Norm_mean: 4.6036e-01
132
+ 2024-06-16 04:25:11,111 INFO: [3644_..][Iter: 160, lr:(7.867e-04,7.867e-06,7.867e-05,)] [eta: 0:11:21] loss: 4.4056e-02 Norm_mean: 4.6365e-01
133
+ 2024-06-16 04:25:22,749 INFO: [3644_..][Iter: 170, lr:(7.733e-04,7.733e-06,7.733e-05,)] [eta: 0:11:10] loss: 2.4251e-01 Norm_mean: 4.6673e-01
134
+ 2024-06-16 04:25:34,503 INFO: [3644_..][Iter: 180, lr:(7.600e-04,7.600e-06,7.600e-05,)] [eta: 0:10:59] loss: 1.3388e+00 Norm_mean: 4.7023e-01
135
+ 2024-06-16 04:25:46,144 INFO: [3644_..][Iter: 190, lr:(7.467e-04,7.467e-06,7.467e-05,)] [eta: 0:10:47] loss: 8.0059e-01 Norm_mean: 4.7438e-01
136
+ 2024-06-16 04:25:57,845 INFO: [3644_..][Iter: 200, lr:(7.333e-04,7.333e-06,7.333e-05,)] [eta: 0:10:36] loss: 6.1990e-01 Norm_mean: 4.7834e-01
137
+ 2024-06-16 04:26:09,571 INFO: [3644_..][Iter: 210, lr:(7.200e-04,7.200e-06,7.200e-05,)] [eta: 0:10:25] loss: 1.2676e+00 Norm_mean: 4.8125e-01
138
+ 2024-06-16 04:26:21,404 INFO: [3644_..][Iter: 220, lr:(7.067e-04,7.067e-06,7.067e-05,)] [eta: 0:10:14] loss: 4.1141e-01 Norm_mean: 4.8385e-01
139
+ 2024-06-16 04:26:33,250 INFO: [3644_..][Iter: 230, lr:(6.933e-04,6.933e-06,6.933e-05,)] [eta: 0:10:03] loss: 4.3060e-01 Norm_mean: 4.8677e-01
140
+ 2024-06-16 04:26:45,122 INFO: [3644_..][Iter: 240, lr:(6.800e-04,6.800e-06,6.800e-05,)] [eta: 0:09:51] loss: 2.2877e+00 Norm_mean: 4.8967e-01
141
+ 2024-06-16 04:26:56,731 INFO: [3644_..][Iter: 250, lr:(6.667e-04,6.667e-06,6.667e-05,)] [eta: 0:09:40] loss: 2.6713e-01 Norm_mean: 4.9255e-01
142
+ 2024-06-16 04:27:08,419 INFO: [3644_..][Iter: 260, lr:(6.533e-04,6.533e-06,6.533e-05,)] [eta: 0:09:28] loss: 3.4979e-01 Norm_mean: 4.9546e-01
143
+ 2024-06-16 04:27:20,150 INFO: [3644_..][Iter: 270, lr:(6.400e-04,6.400e-06,6.400e-05,)] [eta: 0:09:17] loss: 1.0274e-01 Norm_mean: 4.9865e-01
144
+ 2024-06-16 04:27:31,895 INFO: [3644_..][Iter: 280, lr:(6.267e-04,6.267e-06,6.267e-05,)] [eta: 0:09:05] loss: 2.1467e-01 Norm_mean: 5.0172e-01
145
+ 2024-06-16 04:27:43,536 INFO: [3644_..][Iter: 290, lr:(6.133e-04,6.133e-06,6.133e-05,)] [eta: 0:08:54] loss: 1.4074e-01 Norm_mean: 5.0442e-01
146
+ 2024-06-16 04:27:55,450 INFO: [3644_..][Iter: 300, lr:(6.000e-04,6.000e-06,6.000e-05,)] [eta: 0:08:42] loss: 1.9407e-01 Norm_mean: 5.0664e-01
147
+ 2024-06-16 04:28:07,265 INFO: [3644_..][Iter: 310, lr:(5.867e-04,5.867e-06,5.867e-05,)] [eta: 0:08:31] loss: 9.7688e-02 Norm_mean: 5.0850e-01
148
+ 2024-06-16 04:28:19,112 INFO: [3644_..][Iter: 320, lr:(5.733e-04,5.733e-06,5.733e-05,)] [eta: 0:08:20] loss: 1.4610e+00 Norm_mean: 5.1087e-01
149
+ 2024-06-16 04:28:30,965 INFO: [3644_..][Iter: 330, lr:(5.600e-04,5.600e-06,5.600e-05,)] [eta: 0:08:08] loss: 4.5426e-01 Norm_mean: 5.1334e-01
150
+ 2024-06-16 04:28:42,756 INFO: [3644_..][Iter: 340, lr:(5.467e-04,5.467e-06,5.467e-05,)] [eta: 0:07:57] loss: 2.7905e-01 Norm_mean: 5.1516e-01
151
+ 2024-06-16 04:28:54,486 INFO: [3644_..][Iter: 350, lr:(5.333e-04,5.333e-06,5.333e-05,)] [eta: 0:07:45] loss: 7.6693e-01 Norm_mean: 5.1676e-01
152
+ 2024-06-16 04:29:06,333 INFO: [3644_..][Iter: 360, lr:(5.200e-04,5.200e-06,5.200e-05,)] [eta: 0:07:34] loss: 7.9797e-02 Norm_mean: 5.1820e-01
153
+ 2024-06-16 04:29:18,141 INFO: [3644_..][Iter: 370, lr:(5.067e-04,5.067e-06,5.067e-05,)] [eta: 0:07:22] loss: 5.2179e-01 Norm_mean: 5.1957e-01
154
+ 2024-06-16 04:29:30,032 INFO: [3644_..][Iter: 380, lr:(4.933e-04,4.933e-06,4.933e-05,)] [eta: 0:07:11] loss: 2.9494e-01 Norm_mean: 5.2115e-01
155
+ 2024-06-16 04:29:41,799 INFO: [3644_..][Iter: 390, lr:(4.800e-04,4.800e-06,4.800e-05,)] [eta: 0:06:59] loss: 6.3320e-02 Norm_mean: 5.2243e-01
156
+ 2024-06-16 04:29:53,479 INFO: [3644_..][Iter: 400, lr:(4.667e-04,4.667e-06,4.667e-05,)] [eta: 0:06:47] loss: 7.0039e-01 Norm_mean: 5.2346e-01
157
+ 2024-06-16 04:30:05,284 INFO: [3644_..][Iter: 410, lr:(4.533e-04,4.533e-06,4.533e-05,)] [eta: 0:06:36] loss: 2.3866e-02 Norm_mean: 5.2482e-01
158
+ 2024-06-16 04:30:17,183 INFO: [3644_..][Iter: 420, lr:(4.400e-04,4.400e-06,4.400e-05,)] [eta: 0:06:24] loss: 1.9908e-01 Norm_mean: 5.2609e-01
159
+ 2024-06-16 04:30:28,910 INFO: [3644_..][Iter: 430, lr:(4.267e-04,4.267e-06,4.267e-05,)] [eta: 0:06:13] loss: 6.4795e-01 Norm_mean: 5.2751e-01
160
+ 2024-06-16 04:30:40,834 INFO: [3644_..][Iter: 440, lr:(4.133e-04,4.133e-06,4.133e-05,)] [eta: 0:06:01] loss: 1.1585e+00 Norm_mean: 5.2908e-01
161
+ 2024-06-16 04:30:52,648 INFO: [3644_..][Iter: 450, lr:(4.000e-04,4.000e-06,4.000e-05,)] [eta: 0:05:49] loss: 1.0454e-01 Norm_mean: 5.3045e-01
162
+ 2024-06-16 04:31:04,552 INFO: [3644_..][Iter: 460, lr:(3.867e-04,3.867e-06,3.867e-05,)] [eta: 0:05:38] loss: 8.7873e-02 Norm_mean: 5.3178e-01
163
+ 2024-06-16 04:31:16,375 INFO: [3644_..][Iter: 470, lr:(3.733e-04,3.733e-06,3.733e-05,)] [eta: 0:05:26] loss: 8.9999e-01 Norm_mean: 5.3296e-01
164
+ 2024-06-16 04:31:28,237 INFO: [3644_..][Iter: 480, lr:(3.600e-04,3.600e-06,3.600e-05,)] [eta: 0:05:15] loss: 1.2997e+00 Norm_mean: 5.3408e-01
165
+ 2024-06-16 04:31:40,054 INFO: [3644_..][Iter: 490, lr:(3.467e-04,3.467e-06,3.467e-05,)] [eta: 0:05:03] loss: 1.1851e+00 Norm_mean: 5.3509e-01
166
+ 2024-06-16 04:31:51,666 INFO: [3644_..][Iter: 500, lr:(3.333e-04,3.333e-06,3.333e-05,)] [eta: 0:04:51] loss: 1.3830e-01 Norm_mean: 5.3596e-01
167
+ 2024-06-16 04:32:03,433 INFO: [3644_..][Iter: 510, lr:(3.200e-04,3.200e-06,3.200e-05,)] [eta: 0:04:39] loss: 4.7774e-01 Norm_mean: 5.3699e-01
168
+ 2024-06-16 04:32:15,289 INFO: [3644_..][Iter: 520, lr:(3.067e-04,3.067e-06,3.067e-05,)] [eta: 0:04:28] loss: 2.6115e-01 Norm_mean: 5.3803e-01
169
+ 2024-06-16 04:32:27,141 INFO: [3644_..][Iter: 530, lr:(2.933e-04,2.933e-06,2.933e-05,)] [eta: 0:04:16] loss: 1.0402e+00 Norm_mean: 5.3886e-01
170
+ 2024-06-16 04:32:38,976 INFO: [3644_..][Iter: 540, lr:(2.800e-04,2.800e-06,2.800e-05,)] [eta: 0:04:04] loss: 2.1990e-01 Norm_mean: 5.3952e-01
171
+ 2024-06-16 04:32:50,663 INFO: [3644_..][Iter: 550, lr:(2.667e-04,2.667e-06,2.667e-05,)] [eta: 0:03:53] loss: 1.4004e-01 Norm_mean: 5.4009e-01
172
+ 2024-06-16 04:33:02,501 INFO: [3644_..][Iter: 560, lr:(2.533e-04,2.533e-06,2.533e-05,)] [eta: 0:03:41] loss: 8.0468e-01 Norm_mean: 5.4064e-01
173
+ 2024-06-16 04:33:14,162 INFO: [3644_..][Iter: 570, lr:(2.400e-04,2.400e-06,2.400e-05,)] [eta: 0:03:29] loss: 6.3702e-02 Norm_mean: 5.4116e-01
174
+ 2024-06-16 04:33:25,913 INFO: [3644_..][Iter: 580, lr:(2.267e-04,2.267e-06,2.267e-05,)] [eta: 0:03:18] loss: 4.4585e-01 Norm_mean: 5.4157e-01
175
+ 2024-06-16 04:33:37,707 INFO: [3644_..][Iter: 590, lr:(2.133e-04,2.133e-06,2.133e-05,)] [eta: 0:03:06] loss: 8.7879e-01 Norm_mean: 5.4198e-01
176
+ 2024-06-16 04:33:49,744 INFO: [3644_..][Iter: 600, lr:(2.000e-04,2.000e-06,2.000e-05,)] [eta: 0:02:54] loss: 7.9852e-01 Norm_mean: 5.4228e-01
177
+ 2024-06-16 04:34:01,609 INFO: [3644_..][Iter: 610, lr:(1.867e-04,1.867e-06,1.867e-05,)] [eta: 0:02:43] loss: 4.8449e-01 Norm_mean: 5.4260e-01
178
+ 2024-06-16 04:34:13,338 INFO: [3644_..][Iter: 620, lr:(1.733e-04,1.733e-06,1.733e-05,)] [eta: 0:02:31] loss: 4.7763e-01 Norm_mean: 5.4294e-01
179
+ 2024-06-16 04:34:25,185 INFO: [3644_..][Iter: 630, lr:(1.600e-04,1.600e-06,1.600e-05,)] [eta: 0:02:19] loss: 1.8500e-01 Norm_mean: 5.4322e-01
180
+ 2024-06-16 04:34:37,064 INFO: [3644_..][Iter: 640, lr:(1.467e-04,1.467e-06,1.467e-05,)] [eta: 0:02:07] loss: 3.4307e-01 Norm_mean: 5.4342e-01
181
+ 2024-06-16 04:34:49,028 INFO: [3644_..][Iter: 650, lr:(1.333e-04,1.333e-06,1.333e-05,)] [eta: 0:01:56] loss: 3.6797e-01 Norm_mean: 5.4367e-01
182
+ 2024-06-16 04:35:00,772 INFO: [3644_..][Iter: 660, lr:(1.200e-04,1.200e-06,1.200e-05,)] [eta: 0:01:44] loss: 5.8958e-01 Norm_mean: 5.4389e-01
183
+ 2024-06-16 04:35:12,700 INFO: [3644_..][Iter: 670, lr:(1.067e-04,1.067e-06,1.067e-05,)] [eta: 0:01:32] loss: 3.1125e-01 Norm_mean: 5.4404e-01
184
+ 2024-06-16 04:35:25,252 INFO: [3644_..][Iter: 680, lr:(9.333e-05,9.333e-07,9.333e-06,)] [eta: 0:01:21] loss: 1.1720e+00 Norm_mean: 5.4415e-01
185
+ 2024-06-16 04:35:37,292 INFO: [3644_..][Iter: 690, lr:(8.000e-05,8.000e-07,8.000e-06,)] [eta: 0:01:09] loss: 1.0099e-01 Norm_mean: 5.4423e-01
186
+ 2024-06-16 04:35:49,078 INFO: [3644_..][Iter: 700, lr:(6.667e-05,6.667e-07,6.667e-06,)] [eta: 0:00:57] loss: 7.8378e-01 Norm_mean: 5.4430e-01
187
+ 2024-06-16 04:36:00,940 INFO: [3644_..][Iter: 710, lr:(5.333e-05,5.333e-07,5.333e-06,)] [eta: 0:00:45] loss: 2.3071e-01 Norm_mean: 5.4434e-01
188
+ 2024-06-16 04:36:12,840 INFO: [3644_..][Iter: 720, lr:(4.000e-05,4.000e-07,4.000e-06,)] [eta: 0:00:34] loss: 2.5471e-02 Norm_mean: 5.4436e-01
189
+ 2024-06-16 04:36:24,815 INFO: [3644_..][Iter: 730, lr:(2.667e-05,2.667e-07,2.667e-06,)] [eta: 0:00:22] loss: 3.7025e-01 Norm_mean: 5.4438e-01
190
+ 2024-06-16 04:36:36,631 INFO: [3644_..][Iter: 740, lr:(1.333e-05,1.333e-07,1.333e-06,)] [eta: 0:00:10] loss: 9.3596e-02 Norm_mean: 5.4439e-01
191
+ 2024-06-16 04:36:48,366 INFO: [3644_..][Iter: 750, lr:(0.000e+00,0.000e+00,0.000e+00,)] [eta: -1 day, 23:59:59] loss: 9.7311e-02 Norm_mean: 5.4439e-01
192
+ 2024-06-16 04:36:48,496 INFO: Save state to /home/ujinsong/workspace/ortha/experiments/3644_miguel_ortho/models/edlora_model-latest.pth
experiments/single-concept/mirabel/1058_mirabel_ortho.yml ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # GENERATE TIME: Sun Jun 16 05:15:43 2024
2
+ # CMD:
3
+ # train_edlora.py -opt single-concept/train_configs/1058_mirabel_ortho.yml
4
+
5
+ name: 1058_mirabel_ortho
6
+ manual_seed: 1058
7
+ mixed_precision: fp16
8
+ gradient_accumulation_steps: 1
9
+ datasets:
10
+ train:
11
+ name: LoraDataset
12
+ concept_list: single-concept/data_configs/mirabel.json
13
+ use_caption: true
14
+ use_mask: true
15
+ instance_transform:
16
+ - type: HumanResizeCropFinalV3
17
+ size: 512
18
+ crop_p: 0.5
19
+ - type: ToTensor
20
+ - type: Normalize
21
+ mean:
22
+ - 0.5
23
+ std:
24
+ - 0.5
25
+ - type: ShuffleCaption
26
+ keep_token_num: 1
27
+ - type: EnhanceText
28
+ enhance_type: human
29
+ replace_mapping:
30
+ <TOK>: <mirabel1> <mirabel2>
31
+ batch_size_per_gpu: 2
32
+ dataset_enlarge_ratio: 500
33
+ val_vis:
34
+ name: PromptDataset
35
+ prompts: single-concept/validation_prompts/characters/test_girl_disney.txt
36
+ num_samples_per_prompt: 8
37
+ latent_size:
38
+ - 4
39
+ - 64
40
+ - 64
41
+ replace_mapping:
42
+ <TOK>: <mirabel1> <mirabel2>
43
+ batch_size_per_gpu: 4
44
+ models:
45
+ pretrained_path: nitrosocke/mo-di-diffusion
46
+ enable_edlora: true
47
+ finetune_cfg:
48
+ text_embedding:
49
+ enable_tuning: true
50
+ lr: 0.001
51
+ text_encoder:
52
+ enable_tuning: true
53
+ lora_cfg:
54
+ rank: 5
55
+ alpha: 1.0
56
+ where: CLIPAttention
57
+ lr: 1.0e-05
58
+ unet:
59
+ enable_tuning: true
60
+ lora_cfg:
61
+ rank: 5
62
+ alpha: 1.0
63
+ where: Attention
64
+ lr: 0.0001
65
+ new_concept_token: <mirabel1>+<mirabel2>
66
+ initializer_token: <rand-0.013>+girl
67
+ noise_offset: 0.01
68
+ attn_reg_weight: 0.01
69
+ reg_full_identity: false
70
+ use_mask_loss: true
71
+ gradient_checkpoint: false
72
+ enable_xformers: true
73
+ path:
74
+ pretrain_network: null
75
+ train:
76
+ optim_g:
77
+ type: AdamW
78
+ lr: 0.0
79
+ weight_decay: 0.01
80
+ betas:
81
+ - 0.9
82
+ - 0.999
83
+ unet_kv_drop_rate: 0
84
+ scheduler: linear
85
+ emb_norm_threshold: 0.55
86
+ val:
87
+ val_during_save: false
88
+ compose_visualize: false
89
+ alpha_list:
90
+ - 0
91
+ - 0.7
92
+ - 1.0
93
+ sample:
94
+ num_inference_steps: 50
95
+ guidance_scale: 7.5
96
+ logger:
97
+ print_freq: 10
98
+ save_checkpoint_freq: 10000.0
experiments/single-concept/mirabel/models/edlora_model-latest.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e64b82ae3b8750437fc33441ec3f6018790f0f496b73c706ead9b42e1ae1698d
3
+ size 35173046
experiments/single-concept/mirabel/train_1058_mirabel_ortho_20240616_051543.log ADDED
@@ -0,0 +1,192 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2024-06-16 05:15:43,407 INFO: Distributed environment: MULTI_GPU Backend: nccl
2
+ Num processes: 2
3
+ Process index: 0
4
+ Local process index: 0
5
+ Device: cuda:0
6
+
7
+ Mixed precision type: fp16
8
+
9
+ 2024-06-16 05:15:43,411 INFO:
10
+ name: 1058_mirabel_ortho
11
+ manual_seed: 1058
12
+ mixed_precision: fp16
13
+ gradient_accumulation_steps: 1
14
+ datasets:[
15
+ train:[
16
+ name: LoraDataset
17
+ concept_list: single-concept/data_configs/mirabel.json
18
+ use_caption: True
19
+ use_mask: True
20
+ instance_transform: [{'type': 'HumanResizeCropFinalV3', 'size': 512, 'crop_p': 0.5}, {'type': 'ToTensor'}, {'type': 'Normalize', 'mean': [0.5], 'std': [0.5]}, {'type': 'ShuffleCaption', 'keep_token_num': 1}, {'type': 'EnhanceText', 'enhance_type': 'human'}]
21
+ replace_mapping:[
22
+ <TOK>: <mirabel1> <mirabel2>
23
+ ]
24
+ batch_size_per_gpu: 2
25
+ dataset_enlarge_ratio: 500
26
+ ]
27
+ val_vis:[
28
+ name: PromptDataset
29
+ prompts: single-concept/validation_prompts/characters/test_girl_disney.txt
30
+ num_samples_per_prompt: 8
31
+ latent_size: [4, 64, 64]
32
+ replace_mapping:[
33
+ <TOK>: <mirabel1> <mirabel2>
34
+ ]
35
+ batch_size_per_gpu: 4
36
+ ]
37
+ ]
38
+ models:[
39
+ pretrained_path: nitrosocke/mo-di-diffusion
40
+ enable_edlora: True
41
+ finetune_cfg:[
42
+ text_embedding:[
43
+ enable_tuning: True
44
+ lr: 0.001
45
+ ]
46
+ text_encoder:[
47
+ enable_tuning: True
48
+ lora_cfg:[
49
+ rank: 5
50
+ alpha: 1.0
51
+ where: CLIPAttention
52
+ ]
53
+ lr: 1e-05
54
+ ]
55
+ unet:[
56
+ enable_tuning: True
57
+ lora_cfg:[
58
+ rank: 5
59
+ alpha: 1.0
60
+ where: Attention
61
+ ]
62
+ lr: 0.0001
63
+ ]
64
+ ]
65
+ new_concept_token: <mirabel1>+<mirabel2>
66
+ initializer_token: <rand-0.013>+girl
67
+ noise_offset: 0.01
68
+ attn_reg_weight: 0.01
69
+ reg_full_identity: False
70
+ use_mask_loss: True
71
+ gradient_checkpoint: False
72
+ enable_xformers: True
73
+ ]
74
+ path:[
75
+ pretrain_network: None
76
+ experiments_root: /home/ujinsong/workspace/ortha/experiments/1058_mirabel_ortho
77
+ models: /home/ujinsong/workspace/ortha/experiments/1058_mirabel_ortho/models
78
+ log: /home/ujinsong/workspace/ortha/experiments/1058_mirabel_ortho
79
+ visualization: /home/ujinsong/workspace/ortha/experiments/1058_mirabel_ortho/visualization
80
+ ]
81
+ train:[
82
+ optim_g:[
83
+ type: AdamW
84
+ lr: 0.0
85
+ weight_decay: 0.01
86
+ betas: [0.9, 0.999]
87
+ ]
88
+ unet_kv_drop_rate: 0
89
+ scheduler: linear
90
+ emb_norm_threshold: 0.55
91
+ ]
92
+ val:[
93
+ val_during_save: False
94
+ compose_visualize: False
95
+ alpha_list: [0, 0.7, 1.0]
96
+ sample:[
97
+ num_inference_steps: 50
98
+ guidance_scale: 7.5
99
+ ]
100
+ ]
101
+ logger:[
102
+ print_freq: 10
103
+ save_checkpoint_freq: 10000.0
104
+ ]
105
+ is_train: True
106
+
107
+ 2024-06-16 05:16:27,718 INFO: <mirabel1> (49408-49423) is random initialized by: <rand-0.013>
108
+ 2024-06-16 05:16:28,351 INFO: <mirabel2> (49424-49439) is random initialized by existing token (girl): 1611
109
+ 2024-06-16 05:16:28,357 INFO: optimizing embedding using lr: 0.001
110
+ 2024-06-16 05:16:28,792 INFO: optimizing text_encoder (48 LoRAs), using lr: 1e-05
111
+ 2024-06-16 05:16:29,899 INFO: optimizing unet (128 LoRAs), using lr: 0.0001
112
+ 2024-06-16 05:16:32,128 INFO: ***** Running training *****
113
+ 2024-06-16 05:16:32,129 INFO: Num examples = 3000
114
+ 2024-06-16 05:16:32,129 INFO: Instantaneous batch size per device = 2
115
+ 2024-06-16 05:16:32,129 INFO: Total train batch size (w. parallel, distributed & accumulation) = 4
116
+ 2024-06-16 05:16:32,129 INFO: Total optimization steps = 750.0
117
+ 2024-06-16 05:17:11,151 INFO: [1058_..][Iter: 10, lr:(9.867e-04,9.867e-06,9.867e-05,)] [eta: 0:43:41] loss: 6.8763e-01 Norm_mean: 3.7366e-01
118
+ 2024-06-16 05:17:23,058 INFO: [1058_..][Iter: 20, lr:(9.733e-04,9.733e-06,9.733e-05,)] [eta: 0:29:27] loss: 6.2487e-01 Norm_mean: 3.9053e-01
119
+ 2024-06-16 05:17:34,798 INFO: [1058_..][Iter: 30, lr:(9.600e-04,9.600e-06,9.600e-05,)] [eta: 0:24:13] loss: 6.2761e-02 Norm_mean: 4.0165e-01
120
+ 2024-06-16 05:17:46,993 INFO: [1058_..][Iter: 40, lr:(9.467e-04,9.467e-06,9.467e-05,)] [eta: 0:21:34] loss: 3.1695e-01 Norm_mean: 4.0899e-01
121
+ 2024-06-16 05:17:59,083 INFO: [1058_..][Iter: 50, lr:(9.333e-04,9.333e-06,9.333e-05,)] [eta: 0:19:51] loss: 7.8568e-01 Norm_mean: 4.1499e-01
122
+ 2024-06-16 05:18:10,819 INFO: [1058_..][Iter: 60, lr:(9.200e-04,9.200e-06,9.200e-05,)] [eta: 0:18:34] loss: 5.5314e-01 Norm_mean: 4.2029e-01
123
+ 2024-06-16 05:18:22,642 INFO: [1058_..][Iter: 70, lr:(9.067e-04,9.067e-06,9.067e-05,)] [eta: 0:17:36] loss: 5.4705e-01 Norm_mean: 4.2650e-01
124
+ 2024-06-16 05:18:34,377 INFO: [1058_..][Iter: 80, lr:(8.933e-04,8.933e-06,8.933e-05,)] [eta: 0:16:49] loss: 5.9230e-01 Norm_mean: 4.3279e-01
125
+ 2024-06-16 05:18:46,315 INFO: [1058_..][Iter: 90, lr:(8.800e-04,8.800e-06,8.800e-05,)] [eta: 0:16:11] loss: 7.2425e-01 Norm_mean: 4.3999e-01
126
+ 2024-06-16 05:18:58,302 INFO: [1058_..][Iter: 100, lr:(8.667e-04,8.667e-06,8.667e-05,)] [eta: 0:15:39] loss: 1.0467e-01 Norm_mean: 4.4887e-01
127
+ 2024-06-16 05:19:10,106 INFO: [1058_..][Iter: 110, lr:(8.533e-04,8.533e-06,8.533e-05,)] [eta: 0:15:09] loss: 1.2473e+00 Norm_mean: 4.5736e-01
128
+ 2024-06-16 05:19:22,032 INFO: [1058_..][Iter: 120, lr:(8.400e-04,8.400e-06,8.400e-05,)] [eta: 0:14:43] loss: 2.2287e-01 Norm_mean: 4.6546e-01
129
+ 2024-06-16 05:19:34,115 INFO: [1058_..][Iter: 130, lr:(8.267e-04,8.267e-06,8.267e-05,)] [eta: 0:14:19] loss: 7.2866e-02 Norm_mean: 4.7126e-01
130
+ 2024-06-16 05:19:46,251 INFO: [1058_..][Iter: 140, lr:(8.133e-04,8.133e-06,8.133e-05,)] [eta: 0:13:58] loss: 2.6349e-01 Norm_mean: 4.7589e-01
131
+ 2024-06-16 05:19:58,306 INFO: [1058_..][Iter: 150, lr:(8.000e-04,8.000e-06,8.000e-05,)] [eta: 0:13:37] loss: 2.1214e+00 Norm_mean: 4.7993e-01
132
+ 2024-06-16 05:20:10,428 INFO: [1058_..][Iter: 160, lr:(7.867e-04,7.867e-06,7.867e-05,)] [eta: 0:13:18] loss: 2.5936e-01 Norm_mean: 4.8352e-01
133
+ 2024-06-16 05:20:22,408 INFO: [1058_..][Iter: 170, lr:(7.733e-04,7.733e-06,7.733e-05,)] [eta: 0:12:59] loss: 1.0543e+00 Norm_mean: 4.8775e-01
134
+ 2024-06-16 05:20:34,285 INFO: [1058_..][Iter: 180, lr:(7.600e-04,7.600e-06,7.600e-05,)] [eta: 0:12:41] loss: 3.1821e-01 Norm_mean: 4.9118e-01
135
+ 2024-06-16 05:20:46,301 INFO: [1058_..][Iter: 190, lr:(7.467e-04,7.467e-06,7.467e-05,)] [eta: 0:12:23] loss: 1.5731e-01 Norm_mean: 4.9415e-01
136
+ 2024-06-16 05:20:58,797 INFO: [1058_..][Iter: 200, lr:(7.333e-04,7.333e-06,7.333e-05,)] [eta: 0:12:08] loss: 5.9038e-01 Norm_mean: 4.9735e-01
137
+ 2024-06-16 05:21:10,685 INFO: [1058_..][Iter: 210, lr:(7.200e-04,7.200e-06,7.200e-05,)] [eta: 0:11:51] loss: 7.3782e-01 Norm_mean: 5.0128e-01
138
+ 2024-06-16 05:21:22,596 INFO: [1058_..][Iter: 220, lr:(7.067e-04,7.067e-06,7.067e-05,)] [eta: 0:11:35] loss: 1.1098e+00 Norm_mean: 5.0493e-01
139
+ 2024-06-16 05:21:34,719 INFO: [1058_..][Iter: 230, lr:(6.933e-04,6.933e-06,6.933e-05,)] [eta: 0:11:19] loss: 6.1245e-01 Norm_mean: 5.0890e-01
140
+ 2024-06-16 05:21:46,570 INFO: [1058_..][Iter: 240, lr:(6.800e-04,6.800e-06,6.800e-05,)] [eta: 0:11:04] loss: 3.0506e-01 Norm_mean: 5.1191e-01
141
+ 2024-06-16 05:21:58,376 INFO: [1058_..][Iter: 250, lr:(6.667e-04,6.667e-06,6.667e-05,)] [eta: 0:10:48] loss: 1.3792e+00 Norm_mean: 5.1521e-01
142
+ 2024-06-16 05:22:10,561 INFO: [1058_..][Iter: 260, lr:(6.533e-04,6.533e-06,6.533e-05,)] [eta: 0:10:34] loss: 1.8070e-01 Norm_mean: 5.1873e-01
143
+ 2024-06-16 05:22:22,744 INFO: [1058_..][Iter: 270, lr:(6.400e-04,6.400e-06,6.400e-05,)] [eta: 0:10:19] loss: 5.4035e-02 Norm_mean: 5.2224e-01
144
+ 2024-06-16 05:22:35,014 INFO: [1058_..][Iter: 280, lr:(6.267e-04,6.267e-06,6.267e-05,)] [eta: 0:10:05] loss: 1.2486e+00 Norm_mean: 5.2536e-01
145
+ 2024-06-16 05:22:46,948 INFO: [1058_..][Iter: 290, lr:(6.133e-04,6.133e-06,6.133e-05,)] [eta: 0:09:51] loss: 1.0587e+00 Norm_mean: 5.2837e-01
146
+ 2024-06-16 05:22:59,011 INFO: [1058_..][Iter: 300, lr:(6.000e-04,6.000e-06,6.000e-05,)] [eta: 0:09:37] loss: 1.2007e+00 Norm_mean: 5.3108e-01
147
+ 2024-06-16 05:23:10,872 INFO: [1058_..][Iter: 310, lr:(5.867e-04,5.867e-06,5.867e-05,)] [eta: 0:09:22] loss: 1.4058e+00 Norm_mean: 5.3306e-01
148
+ 2024-06-16 05:23:22,809 INFO: [1058_..][Iter: 320, lr:(5.733e-04,5.733e-06,5.733e-05,)] [eta: 0:09:08] loss: 9.8557e-01 Norm_mean: 5.3553e-01
149
+ 2024-06-16 05:23:34,926 INFO: [1058_..][Iter: 330, lr:(5.600e-04,5.600e-06,5.600e-05,)] [eta: 0:08:55] loss: 8.7980e-01 Norm_mean: 5.3783e-01
150
+ 2024-06-16 05:23:47,765 INFO: [1058_..][Iter: 340, lr:(5.467e-04,5.467e-06,5.467e-05,)] [eta: 0:08:42] loss: 6.9745e-01 Norm_mean: 5.4009e-01
151
+ 2024-06-16 05:23:59,594 INFO: [1058_..][Iter: 350, lr:(5.333e-04,5.333e-06,5.333e-05,)] [eta: 0:08:28] loss: 5.2154e-01 Norm_mean: 5.4273e-01
152
+ 2024-06-16 05:24:11,471 INFO: [1058_..][Iter: 360, lr:(5.200e-04,5.200e-06,5.200e-05,)] [eta: 0:08:14] loss: 2.0132e+00 Norm_mean: 5.4635e-01
153
+ 2024-06-16 05:24:23,560 INFO: [1058_..][Iter: 370, lr:(5.067e-04,5.067e-06,5.067e-05,)] [eta: 0:08:01] loss: 1.2155e+00 Norm_mean: 5.4875e-01
154
+ 2024-06-16 05:24:35,585 INFO: [1058_..][Iter: 380, lr:(4.933e-04,4.933e-06,4.933e-05,)] [eta: 0:07:48] loss: 1.2496e+00 Norm_mean: 5.5004e-01
155
+ 2024-06-16 05:24:47,563 INFO: [1058_..][Iter: 390, lr:(4.800e-04,4.800e-06,4.800e-05,)] [eta: 0:07:34] loss: 1.0263e+00 Norm_mean: 5.5004e-01
156
+ 2024-06-16 05:24:59,735 INFO: [1058_..][Iter: 400, lr:(4.667e-04,4.667e-06,4.667e-05,)] [eta: 0:07:21] loss: 1.4509e+00 Norm_mean: 5.5004e-01
157
+ 2024-06-16 05:25:13,581 INFO: [1058_..][Iter: 410, lr:(4.533e-04,4.533e-06,4.533e-05,)] [eta: 0:07:10] loss: 1.4596e+00 Norm_mean: 5.5004e-01
158
+ 2024-06-16 05:25:26,324 INFO: [1058_..][Iter: 420, lr:(4.400e-04,4.400e-06,4.400e-05,)] [eta: 0:06:57] loss: 1.6827e-01 Norm_mean: 5.5004e-01
159
+ 2024-06-16 05:25:38,208 INFO: [1058_..][Iter: 430, lr:(4.267e-04,4.267e-06,4.267e-05,)] [eta: 0:06:44] loss: 4.5188e-02 Norm_mean: 5.5004e-01
160
+ 2024-06-16 05:25:50,206 INFO: [1058_..][Iter: 440, lr:(4.133e-04,4.133e-06,4.133e-05,)] [eta: 0:06:31] loss: 3.8552e-01 Norm_mean: 5.5004e-01
161
+ 2024-06-16 05:26:02,213 INFO: [1058_..][Iter: 450, lr:(4.000e-04,4.000e-06,4.000e-05,)] [eta: 0:06:17] loss: 9.3223e-01 Norm_mean: 5.5004e-01
162
+ 2024-06-16 05:26:14,522 INFO: [1058_..][Iter: 460, lr:(3.867e-04,3.867e-06,3.867e-05,)] [eta: 0:06:05] loss: 3.7621e-01 Norm_mean: 5.5004e-01
163
+ 2024-06-16 05:26:26,438 INFO: [1058_..][Iter: 470, lr:(3.733e-04,3.733e-06,3.733e-05,)] [eta: 0:05:52] loss: 6.5765e-01 Norm_mean: 5.5004e-01
164
+ 2024-06-16 05:26:38,281 INFO: [1058_..][Iter: 480, lr:(3.600e-04,3.600e-06,3.600e-05,)] [eta: 0:05:38] loss: 8.8728e-01 Norm_mean: 5.5004e-01
165
+ 2024-06-16 05:26:50,158 INFO: [1058_..][Iter: 490, lr:(3.467e-04,3.467e-06,3.467e-05,)] [eta: 0:05:26] loss: 1.6409e-01 Norm_mean: 5.5004e-01
166
+ 2024-06-16 05:27:01,945 INFO: [1058_..][Iter: 500, lr:(3.333e-04,3.333e-06,3.333e-05,)] [eta: 0:05:13] loss: 1.9812e+00 Norm_mean: 5.5004e-01
167
+ 2024-06-16 05:27:13,699 INFO: [1058_..][Iter: 510, lr:(3.200e-04,3.200e-06,3.200e-05,)] [eta: 0:05:00] loss: 6.0130e-01 Norm_mean: 5.5004e-01
168
+ 2024-06-16 05:27:25,810 INFO: [1058_..][Iter: 520, lr:(3.067e-04,3.067e-06,3.067e-05,)] [eta: 0:04:47] loss: 3.8685e-01 Norm_mean: 5.5004e-01
169
+ 2024-06-16 05:27:37,599 INFO: [1058_..][Iter: 530, lr:(2.933e-04,2.933e-06,2.933e-05,)] [eta: 0:04:34] loss: 1.4051e-01 Norm_mean: 5.5004e-01
170
+ 2024-06-16 05:27:49,427 INFO: [1058_..][Iter: 540, lr:(2.800e-04,2.800e-06,2.800e-05,)] [eta: 0:04:21] loss: 1.3837e+00 Norm_mean: 5.5004e-01
171
+ 2024-06-16 05:28:01,353 INFO: [1058_..][Iter: 550, lr:(2.667e-04,2.667e-06,2.667e-05,)] [eta: 0:04:08] loss: 5.3206e-01 Norm_mean: 5.5004e-01
172
+ 2024-06-16 05:28:13,334 INFO: [1058_..][Iter: 560, lr:(2.533e-04,2.533e-06,2.533e-05,)] [eta: 0:03:56] loss: 8.8792e-02 Norm_mean: 5.5004e-01
173
+ 2024-06-16 05:28:25,262 INFO: [1058_..][Iter: 570, lr:(2.400e-04,2.400e-06,2.400e-05,)] [eta: 0:03:43] loss: 1.7896e+00 Norm_mean: 5.5004e-01
174
+ 2024-06-16 05:28:37,296 INFO: [1058_..][Iter: 580, lr:(2.267e-04,2.267e-06,2.267e-05,)] [eta: 0:03:30] loss: 7.7512e-02 Norm_mean: 5.5004e-01
175
+ 2024-06-16 05:28:49,592 INFO: [1058_..][Iter: 590, lr:(2.133e-04,2.133e-06,2.133e-05,)] [eta: 0:03:18] loss: 8.4157e-02 Norm_mean: 5.5004e-01
176
+ 2024-06-16 05:29:01,500 INFO: [1058_..][Iter: 600, lr:(2.000e-04,2.000e-06,2.000e-05,)] [eta: 0:03:05] loss: 2.5472e+00 Norm_mean: 5.5004e-01
177
+ 2024-06-16 05:29:13,214 INFO: [1058_..][Iter: 610, lr:(1.867e-04,1.867e-06,1.867e-05,)] [eta: 0:02:53] loss: 1.6361e+00 Norm_mean: 5.5004e-01
178
+ 2024-06-16 05:29:25,017 INFO: [1058_..][Iter: 620, lr:(1.733e-04,1.733e-06,1.733e-05,)] [eta: 0:02:40] loss: 3.9784e-01 Norm_mean: 5.5004e-01
179
+ 2024-06-16 05:29:36,927 INFO: [1058_..][Iter: 630, lr:(1.600e-04,1.600e-06,1.600e-05,)] [eta: 0:02:28] loss: 2.2515e-01 Norm_mean: 5.5004e-01
180
+ 2024-06-16 05:29:49,030 INFO: [1058_..][Iter: 640, lr:(1.467e-04,1.467e-06,1.467e-05,)] [eta: 0:02:15] loss: 1.4636e-01 Norm_mean: 5.5004e-01
181
+ 2024-06-16 05:30:01,024 INFO: [1058_..][Iter: 650, lr:(1.333e-04,1.333e-06,1.333e-05,)] [eta: 0:02:03] loss: 3.4580e-01 Norm_mean: 5.5004e-01
182
+ 2024-06-16 05:30:13,029 INFO: [1058_..][Iter: 660, lr:(1.200e-04,1.200e-06,1.200e-05,)] [eta: 0:01:50] loss: 4.7256e-01 Norm_mean: 5.5004e-01
183
+ 2024-06-16 05:30:24,832 INFO: [1058_..][Iter: 670, lr:(1.067e-04,1.067e-06,1.067e-05,)] [eta: 0:01:38] loss: 3.6390e-01 Norm_mean: 5.5004e-01
184
+ 2024-06-16 05:30:36,742 INFO: [1058_..][Iter: 680, lr:(9.333e-05,9.333e-07,9.333e-06,)] [eta: 0:01:25] loss: 3.1876e-01 Norm_mean: 5.5004e-01
185
+ 2024-06-16 05:30:48,721 INFO: [1058_..][Iter: 690, lr:(8.000e-05,8.000e-07,8.000e-06,)] [eta: 0:01:13] loss: 1.1820e+00 Norm_mean: 5.5004e-01
186
+ 2024-06-16 05:31:00,674 INFO: [1058_..][Iter: 700, lr:(6.667e-05,6.667e-07,6.667e-06,)] [eta: 0:01:00] loss: 9.7044e-01 Norm_mean: 5.5004e-01
187
+ 2024-06-16 05:31:12,595 INFO: [1058_..][Iter: 710, lr:(5.333e-05,5.333e-07,5.333e-06,)] [eta: 0:00:48] loss: 8.6750e-01 Norm_mean: 5.5004e-01
188
+ 2024-06-16 05:31:24,606 INFO: [1058_..][Iter: 720, lr:(4.000e-05,4.000e-07,4.000e-06,)] [eta: 0:00:35] loss: 7.4090e-01 Norm_mean: 5.5004e-01
189
+ 2024-06-16 05:31:36,389 INFO: [1058_..][Iter: 730, lr:(2.667e-05,2.667e-07,2.667e-06,)] [eta: 0:00:23] loss: 1.8269e+00 Norm_mean: 5.5004e-01
190
+ 2024-06-16 05:31:48,338 INFO: [1058_..][Iter: 740, lr:(1.333e-05,1.333e-07,1.333e-06,)] [eta: 0:00:11] loss: 2.5793e-01 Norm_mean: 5.5004e-01
191
+ 2024-06-16 05:32:00,151 INFO: [1058_..][Iter: 750, lr:(0.000e+00,0.000e+00,0.000e+00,)] [eta: -1 day, 23:59:59] loss: 7.8143e-01 Norm_mean: 5.5004e-01
192
+ 2024-06-16 05:32:00,239 INFO: Save state to /home/ujinsong/workspace/ortha/experiments/1058_mirabel_ortho/models/edlora_model-latest.pth