GaetanMichelet's picture
End of training
eb97ed2 verified
metadata
base_model: google/gemma-2-2b-it
datasets:
  - GaetanMichelet/chat-60_ft_task-3
  - GaetanMichelet/chat-120_ft_task-3
  - GaetanMichelet/chat-180_ft_task-3
library_name: peft
license: gemma
tags:
  - alignment-handbook
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: Gemma-2-2B_task-3_180-samples_config-2
    results: []

Gemma-2-2B_task-3_180-samples_config-2

This model is a fine-tuned version of google/gemma-2-2b-it on the GaetanMichelet/chat-60_ft_task-3, the GaetanMichelet/chat-120_ft_task-3 and the GaetanMichelet/chat-180_ft_task-3 datasets. It achieves the following results on the evaluation set:

  • Loss: 0.4529

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 50

Training results

Training Loss Epoch Step Validation Loss
3.2784 0.9412 8 3.0451
1.6125 2.0 17 1.3316
0.7031 2.9412 25 0.6673
0.3776 4.0 34 0.5516
0.3705 4.9412 42 0.4893
0.4428 6.0 51 0.4587
0.264 6.9412 59 0.4529
0.2488 8.0 68 0.4680
0.211 8.9412 76 0.4985
0.1326 10.0 85 0.5514
0.1028 10.9412 93 0.6646
0.0306 12.0 102 0.8112
0.0188 12.9412 110 0.9622
0.0227 14.0 119 0.9967

Framework versions

  • PEFT 0.12.0
  • Transformers 4.44.0
  • Pytorch 2.1.2+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1