brevity_gs-promt_e30

This model is a fine-tuned version of TheBloke/Mistral-7B-Instruct-v0.2-GPTQ on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3849

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 2
• num_epochs: 30
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
3.0749	0.8	2	2.2613
1.8691	2.0	5	1.9098
2.4011	2.8	7	1.7179
1.4252	4.0	10	1.4839
1.8772	4.8	12	1.3338
1.0883	6.0	15	1.1161
1.3875	6.8	17	0.9824
0.7889	8.0	20	0.8053
0.9835	8.8	22	0.7143
0.5675	10.0	25	0.6228
0.7408	10.8	27	0.5772
0.4325	12.0	30	0.5213
0.5746	12.8	32	0.4935
0.3531	14.0	35	0.4603
0.4814	14.8	37	0.4431
0.3017	16.0	40	0.4227
0.4213	16.8	42	0.4130
0.2693	18.0	45	0.4021
0.3811	18.8	47	0.3966
0.25	20.0	50	0.3910
0.3578	20.8	52	0.3886
0.2316	22.0	55	0.3863
0.3407	22.8	57	0.3854
0.2228	24.0	60	0.3849

Framework versions

• PEFT 0.10.0
• Transformers 4.40.2
• Pytorch 2.1.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1