phi-3-mini-LoRA

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3597

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.2
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
1.761	0.1684	50	1.8291
1.751	0.3367	100	1.7497
1.6426	0.5051	150	1.6412
1.5228	0.6734	200	1.4889
1.3624	0.8418	250	1.2639
1.126	1.0101	300	1.0570
0.9809	1.1785	350	0.9361
0.8824	1.3468	400	0.8499
0.7793	1.5152	450	0.7608
0.7179	1.6835	500	0.6796
0.6469	1.8519	550	0.6057
0.5654	2.0202	600	0.5418
0.5096	2.1886	650	0.4859
0.4625	2.3569	700	0.4365
0.432	2.5253	750	0.3976
0.3866	2.6936	800	0.3732
0.3725	2.8620	850	0.3597

Framework versions

• PEFT 0.12.0
• Transformers 4.44.2
• Pytorch 2.4.1+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1