Spruteus's picture
Spruteus/brevity_ss-promt_e30
dac6bdf verified
metadata
license: apache-2.0
library_name: peft
tags:
  - generated_from_trainer
base_model: TheBloke/Mistral-7B-Instruct-v0.2-GPTQ
model-index:
  - name: brevity_ss-promt_e30
    results: []

brevity_ss-promt_e30

This model is a fine-tuned version of TheBloke/Mistral-7B-Instruct-v0.2-GPTQ on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.4920

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 2
  • num_epochs: 30
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
3.9164 0.8 2 3.1402
2.3902 2.0 5 2.6152
3.053 2.8 7 2.3220
1.7893 4.0 10 1.9873
2.3626 4.8 12 1.8054
1.3909 6.0 15 1.5493
1.8073 6.8 17 1.3868
1.0338 8.0 20 1.1445
1.2723 8.8 22 0.9763
0.6819 10.0 25 0.7926
0.8371 10.8 27 0.7085
0.4868 12.0 30 0.6329
0.6522 12.8 32 0.5975
0.4052 14.0 35 0.5653
0.5732 14.8 37 0.5482
0.3641 16.0 40 0.5303
0.5239 16.8 42 0.5213
0.3372 18.0 45 0.5101
0.4896 18.8 47 0.5041
0.3174 20.0 50 0.4974
0.4645 20.8 52 0.4949
0.3046 22.0 55 0.4929
0.4505 22.8 57 0.4923
0.2972 24.0 60 0.4920

Framework versions

  • PEFT 0.10.0
  • Transformers 4.40.2
  • Pytorch 2.1.0+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.1