phi2 fine-tuned with forced tokenization and more samples

README.md

@@ -19,6 +19,13 @@ should probably proofread and complete it, then remove this comment. -->
 # phi2_fine_tune_istanbul_rugs
 
 This model is a fine-tuned version of [microsoft/phi-2](https://huggingface.co/microsoft/phi-2) on the generator dataset.
+It achieves the following results on the evaluation set:
+- eval_loss: 0.2468
+- eval_runtime: 107.7441
+- eval_samples_per_second: 4.093
+- eval_steps_per_second: 0.52
+- epoch: 8.69
+- step: 120
 
 ## Model description
 
@@ -46,17 +53,13 @@ The following hyperparameters were used during training:
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_ratio: 0.05
-- lr_scheduler_warmup_steps: 10
-- num_epochs: 40
-
-### Training results
-
-
+- lr_scheduler_warmup_steps: 5
+- training_steps: 250
 
 ### Framework versions
 
 - PEFT 0.8.2
 - Transformers 4.37.2
 - Pytorch 2.1.0+cu121
-- Datasets 2.17.0
+- Datasets 2.17.1
 - Tokenizers 0.15.2

adapter_config.json

@@ -19,9 +19,9 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
+    "out_proj",
     "fc2",
-    "Wqkv",
-    "out_proj"
+    "Wqkv"
   ],
   "task_type": "CAUSAL_LM",
   "use_rslora": false

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aee9a4a251ad80819f1e170c46c1d78c9228ac1e04c2bf46b9d22fcda00f28b6
+oid sha256:2ddeee7dea06e7186d025d3ff23e630448a046427e25a233ce8eaefd536664d5
 size 115368408

runs/Feb20_01-12-45_4ed94eaf6945/events.out.tfevents.1708391575.4ed94eaf6945.3023.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:721b032d156fccf95b7be72a8b4ce81e5fcbfed1722bdecd80b2350090b0e884
+size 10170

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6e70a0e9f432cc658c80dc5e02e6b70d4d17fe4b58e038b86d01eb55d328fa80
+oid sha256:37f0ae867f1cdd45eb8a79a92d9d9989dc3740ed88a404686e29e27d1a20893e
 size 4728