Hugging Face's logo

GaetanMichelet
/
Llama-31-8B_task-2_180-samples_config-4

PEFT
TensorBoard
Safetensors
llama
alignment-handbook
trl
sft
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Metrics Community
Llama-31-8B_task-2_180-samples_config-4 / README.md
GaetanMichelet's picture
GaetanMichelet
End of training
19bbdf1 verified
preview code
|
raw
history blame contribute delete
No virus
3.67 kB
metadata
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
datasets:
  - GaetanMichelet/chat-60_ft_task-2
  - GaetanMichelet/chat-120_ft_task-2
  - GaetanMichelet/chat-180_ft_task-2
library_name: peft
license: llama3.1
tags:
  - alignment-handbook
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: Llama-31-8B_task-2_180-samples_config-4
    results: []

Llama-31-8B_task-2_180-samples_config-4

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-2, the GaetanMichelet/chat-120_ft_task-2 and the GaetanMichelet/chat-180_ft_task-2 datasets. It achieves the following results on the evaluation set:

  • Loss: 0.7156

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 150

Training results

Training Loss Epoch Step Validation Loss
1.0917 0.9412 8 1.1326
1.1392 2.0 17 1.1271
1.1572 2.9412 25 1.1161
1.0856 4.0 34 1.0970
1.0789 4.9412 42 1.0719
1.0546 6.0 51 1.0337
0.9806 6.9412 59 0.9923
0.9617 8.0 68 0.9472
0.933 8.9412 76 0.9129
0.9005 10.0 85 0.8825
0.9183 10.9412 93 0.8578
0.7739 12.0 102 0.8335
0.7649 12.9412 110 0.8165
0.8197 14.0 119 0.8009
0.7488 14.9412 127 0.7889
0.7651 16.0 136 0.7770
0.6992 16.9412 144 0.7679
0.7609 18.0 153 0.7580
0.6868 18.9412 161 0.7510
0.7077 20.0 170 0.7437
0.6862 20.9412 178 0.7379
0.6939 22.0 187 0.7319
0.6564 22.9412 195 0.7275
0.6446 24.0 204 0.7236
0.6304 24.9412 212 0.7200
0.6583 26.0 221 0.7178
0.5974 26.9412 229 0.7159
0.6266 28.0 238 0.7159
0.6081 28.9412 246 0.7156
0.5853 30.0 255 0.7165
0.54 30.9412 263 0.7200
0.5438 32.0 272 0.7231
0.5163 32.9412 280 0.7268
0.5302 34.0 289 0.7345
0.4865 34.9412 297 0.7542
0.4774 36.0 306 0.7611

Framework versions

  • PEFT 0.12.0
  • Transformers 4.44.0
  • Pytorch 2.1.2+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1