Spruteus's picture
Spruteus/qwen_30B_brevity_tuned
a8f3bd5 verified
metadata
license: other
library_name: peft
tags:
  - trl
  - sft
  - generated_from_trainer
base_model: Qwen/Qwen1.5-32B-Chat
model-index:
  - name: qwen_30B_brevity_tuned
    results: []

qwen_30B_brevity_tuned

This model is a fine-tuned version of Qwen/Qwen1.5-32B-Chat on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.0879

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 30
  • training_steps: 300

Training results

Training Loss Epoch Step Validation Loss
3.2559 1.0 10 3.1422
3.0615 2.0 20 2.8362
2.6919 3.0 30 2.4366
2.2914 4.0 40 2.0373
1.8444 5.0 50 1.5040
1.2158 6.0 60 0.8004
0.568 7.0 70 0.3102
0.2405 8.0 80 0.1604
0.1385 9.0 90 0.1139
0.1012 10.0 100 0.0966
0.0828 11.0 110 0.0920
0.0742 12.0 120 0.0918
0.0652 13.0 130 0.0914
0.0584 14.0 140 0.0906
0.05 15.0 150 0.0891
0.0439 16.0 160 0.0885
0.0368 17.0 170 0.0880
0.0317 18.0 180 0.0866
0.0256 19.0 190 0.0857
0.0213 20.0 200 0.0850
0.0176 21.0 210 0.0849
0.0153 22.0 220 0.0850
0.0136 23.0 230 0.0855
0.0124 24.0 240 0.0860
0.0118 25.0 250 0.0866
0.0113 26.0 260 0.0870
0.0109 27.0 270 0.0874
0.0107 28.0 280 0.0878
0.0103 29.0 290 0.0879
0.0101 30.0 300 0.0879

Framework versions

  • PEFT 0.10.0
  • Transformers 4.40.2
  • Pytorch 2.3.0+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.1