---
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.1
tags:
- generated_from_trainer
model-index:
- name: mistral-alpaca2k-3e
  results: []
---

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
# mistral-alpaca2k-3e

This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the mhenrichsen/alpaca_2k_test dataset.
It achieves the following results on the evaluation set:
- Loss: 0.8850

## Training procedure
accelerate launch -m axolotl.cli.train examples/mistral/qlora.yml

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.392         | 0.0   | 1    | 1.2581          |
| 0.912         | 0.15  | 36   | 0.7686          |
| 0.7114        | 0.3   | 72   | 0.7590          |
| 0.7849        | 0.45  | 108  | 0.7561          |
| 0.693         | 0.61  | 144  | 0.7546          |
| 0.686         | 0.76  | 180  | 0.7538          |
| 0.782         | 0.91  | 216  | 0.7524          |
| 0.5691        | 1.06  | 252  | 0.7700          |
| 0.5295        | 1.21  | 288  | 0.7883          |
| 0.5313        | 1.36  | 324  | 0.7876          |
| 0.4994        | 1.52  | 360  | 0.7971          |
| 0.6007        | 1.67  | 396  | 0.7881          |
| 0.5459        | 1.82  | 432  | 0.7911          |
| 0.5194        | 1.97  | 468  | 0.7924          |
| 0.3376        | 2.12  | 504  | 0.8711          |
| 0.2983        | 2.27  | 540  | 0.8916          |
| 0.341         | 2.43  | 576  | 0.8891          |
| 0.2961        | 2.58  | 612  | 0.8861          |
| 0.2469        | 2.73  | 648  | 0.8860          |
| 0.3535        | 2.88  | 684  | 0.8850          |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.0.1+cu118
- Datasets 2.15.0
- Tokenizers 0.15.0