Gemma_ei_oc_mixed_train

This model is a fine-tuned version of google/gemma-7b on the emollms_ei_oc_mixed dataset. It achieves the following results on the evaluation set:

• Loss: 0.1386

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 32
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss
3.0842	0.3604	10	0.1794
0.1517	0.7207	20	0.1387
0.1383	1.0811	30	0.1458
0.1373	1.4414	40	0.1386
0.1368	1.8018	50	0.1395
0.1379	2.1622	60	0.1398
0.1354	2.5225	70	0.1388

Framework versions

• PEFT 0.11.1
• Transformers 4.41.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1