Finetune-test4 / README.md
AmaanUsmani's picture
AmaanUsmani/Finetune-test4
694e54e verified
metadata
license: apache-2.0
library_name: peft
tags:
  - generated_from_trainer
base_model: TheBloke/Mistral-7B-Instruct-v0.2-GPTQ
model-index:
  - name: Finetune-test4
    results: []

Finetune-test4

This model is a fine-tuned version of TheBloke/Mistral-7B-Instruct-v0.2-GPTQ on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 1.1223

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 2
  • num_epochs: 20
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
0.767 0.9956 56 0.5333
0.4313 1.9911 112 0.4449
0.3107 2.9867 168 0.4640
0.2198 4.0 225 0.5196
0.1633 4.9956 281 0.5811
0.1209 5.9911 337 0.6468
0.0944 6.9867 393 0.6891
0.0745 8.0 450 0.7297
0.064 8.9956 506 0.7844
0.0557 9.9911 562 0.8384
0.0489 10.9867 618 0.8632
0.0433 12.0 675 0.9223
0.0413 12.9956 731 0.9526
0.0389 13.9911 787 0.9552
0.0375 14.9867 843 1.0303
0.0355 16.0 900 1.0489
0.0355 16.9956 956 1.0804
0.0347 17.9911 1012 1.0983
0.0341 18.9867 1068 1.1147
0.0328 19.9111 1120 1.1223

Framework versions

  • PEFT 0.10.0
  • Transformers 4.40.1
  • Pytorch 2.0.1+cu118
  • Datasets 2.19.0
  • Tokenizers 0.19.1