G0514HMA11H

This model is a fine-tuned version of google/gemma-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -17.7793

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.9907	0.09	10	0.2549
-0.3452	0.18	20	-1.4911
-2.6745	0.27	30	-4.3029
-5.5459	0.36	40	-7.2444
-8.4583	0.45	50	-10.1413
-11.1772	0.54	60	-12.6543
-13.5601	0.63	70	-14.6708
-15.3021	0.73	80	-16.0377
-16.4202	0.82	90	-16.8732
-17.0486	0.91	100	-17.2545
-17.3372	1.0	110	-17.4412
-17.4961	1.09	120	-17.5444
-17.5738	1.18	130	-17.6050
-17.6197	1.27	140	-17.6459
-17.6553	1.36	150	-17.6713
-17.6808	1.45	160	-17.6919
-17.6964	1.54	170	-17.7067
-17.7138	1.63	180	-17.7237
-17.7268	1.72	190	-17.7349
-17.7396	1.81	200	-17.7434
-17.7455	1.9	210	-17.7509
-17.7517	1.99	220	-17.7574
-17.7593	2.08	230	-17.7623
-17.7681	2.18	240	-17.7664
-17.7669	2.27	250	-17.7696
-17.7705	2.36	260	-17.7729
-17.7774	2.45	270	-17.7750
-17.7756	2.54	280	-17.7766
-17.7797	2.63	290	-17.7777
-17.7804	2.72	300	-17.7787
-17.7775	2.81	310	-17.7792
-17.7822	2.9	320	-17.7793
-17.7813	2.99	330	-17.7793

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0