PEFT
code
instruct
mistral
souvik0306's picture
Update README.md
15a07f5
metadata
library_name: peft
tags:
  - code
  - instruct
  - mistral
datasets:
  - cognitivecomputations/dolphin-coder
base_model: tiiuae/falcon-7b
license: apache-2.0

Finetuning Overview:

Model Used: tiiuae/falcon-7b

Dataset: cognitivecomputations/dolphin-coder

Dataset Insights:

Dolphin-Coder dataset – a high-quality collection of 100,000+ coding questions and responses. It's perfect for supervised fine-tuning (SFT), and teaching language models to improve on coding-based tasks.

Finetuning Details:

With the utilization of MonsterAPI's no-code LLM finetuner, this finetuning:

  • Was achieved with great cost-effectiveness.
  • Completed in a total duration of 14hr 10mins for 1 epochs using an A6000 48GB GPU.
  • Costed $28.48 for the entire 1 epoch.

Hyperparameters & Additional Details:

  • Epochs: 1
  • Cost Per Epoch: $31.51
  • Model Path: tiiuae/falcon-7b
  • Learning Rate: 0.0002
  • Data Split: 100% train
  • Gradient Accumulation Steps: 128
  • lora r: 32
  • lora alpha: 64

Train Loss


license: apache-2.0