Adriana213's picture
Update README.md
0f967c1 verified
metadata
base_model: openaccess-ai-collective/tiny-mistral
library_name: peft
tags:
  - generated_from_trainer
  - fine-tuning
  - text-generation
model-index:
  - name: tiny-mistral-alpaca-finance
    results: []
datasets:
  - gbharti/finance-alpaca

Tiny Mistral fine-tuned on finance dataset

This model is a fine-tuned version of the openaccess-ai-collective/tiny-mistral language model. It has been fine-tuned on a specialized finance dataset using Parameter-Efficient Fine-Tuning (PEFT) with Low-Rank Adaptation (LoRA). The model is designed to generate responses based on financial instructions and contexts.

Intended uses & limitations

This model is intended for text generation tasks specifically related to financial instructions and contexts. It can be used for generating responses based on given financial prompts.

Limitations:

  • The model may not perform well on financial topics not covered in the training data.
  • The quality of responses may vary depending on the specificity and complexity of the financial queries.
  • The model may generate responses that are not factually accurate or may include biases present in the training data.

Training and evaluation data

The model was fine-tuned on the gbharti/finance-alpaca dataset, which includes financial instructions and outputs. The dataset was processed to format instructions with or without additional context.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 4
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 100
  • num_epochs: 2
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
1.3155 0.2580 500 1.3207
1.1306 0.5160 1000 1.1318
0.9935 0.7739 1500 0.9970
0.7188 1.0319 2000 0.8934
0.6962 1.2899 2500 0.8238
0.6427 1.5479 3000 0.7610
0.6014 1.8059 3500 0.7193

Framework versions

  • PEFT 0.12.0
  • Transformers 4.42.4
  • Pytorch 2.3.1+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1