|
--- |
|
license: openrail |
|
datasets: |
|
- lucasmccabe-lmi/CodeAlpaca-20k |
|
language: |
|
- en |
|
library_name: adapter-transformers |
|
--- |
|
|
|
# Model Card for `opt350m-codealpaca20k` |
|
|
|
## Model Description |
|
|
|
An opt-350m model trained on the CodeAlpaca 20k dataset using quantization and Progressive Embedding Fine-Tuning (PEFT). |
|
The resulting model is designed to understand and generate code-related responses based on the prompts provided. |
|
|
|
[original model car](https://huggingface.co/facebook/opt-350m) |
|
|
|
### Model Architecture |
|
|
|
- **Base Model**: `facebook/opt-350m` |
|
- **Fine-tuning**: Parameter-Efficient Fine-Tuning (PEFT) |
|
|
|
## Training Data |
|
|
|
The model was trained on the `lucasmccabe-lmi/CodeAlpaca-20k` dataset. This dataset contains code-related prompts and their corresponding outputs. |
|
Script used for training is avaiable [here](https://github.com/harpomaxx/llm-finetuning/blob/0954a7ca16bb25bdef6ee9dd1089867bd4d8e0a5/code/python/scripts/stf_train_opt350m.py) |
|
|
|
## Training Procedure |
|
|
|
### Quantization Configuration: |
|
|
|
- **Quantization Type**: 4-bit |
|
- **Compute Dtype**: float16 |
|
- **Double Quant**: Enabled |
|
|
|
### PEFT Configuration: |
|
|
|
- **Lora Alpha**: 16 |
|
- **Lora Dropout**: 0.5 |
|
- **Bias**: None |
|
- **Task Type**: CAUSAL_LM |
|
- **Target Modules**: q_proj, v_proj, k_proj |
|
|
|
### Training Arguments: |
|
|
|
- **Batch Size**: 4 (per device) |
|
- **Gradient Accumulation Steps**: 2 |
|
- **Number of Epochs**: 10 |
|
- **Optimizer**: `adamw_bnb_8bit` |
|
- **Learning Rate**: 2e-5 |
|
- **Max Gradient Norm**: 0.3 |
|
- **Warmup Ratio**: 0.03 |
|
- **Learning Rate Scheduler**: Cosine |
|
- **Logging Steps**: 10 |
|
- **Save Steps**: 250 |
|
- **FP16 Precision**: Enabled |
|
|
|
|
|
### Training information from wandb |
|
|
|
- **train/total_flos:** 72,761,086,854,758,400 |
|
- **train/train_loss:** 1.5557164267259171 |
|
- **train/train_runtime:** 5892.7285 seconds |
|
- **train/train_steps_per_second:** 4.248 |
|
- **_runtime:** 5891.33976650238 seconds |
|
- **_timestamp:** 1,695,390,058.0198596 |
|
- **train/epoch:** 10 |
|
- **train/global_step:** 25,030 |
|
- **train/learning_rate:** 8.371592860045851e-12 |
|
- **train/train_samples_per_second:** 33.977 |
|
- **_step:** 2,503 |
|
- **_wandb.runtime:** 5890 seconds |
|
- **train/loss:** 1.4114 |
|
|
|
|
|
|
|
|
|
|
|
|
|
## Usage |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("facebook/opt350m") |
|
model = AutoModelForCausalLM.from_pretrained("harpomaxx/opt350m-codealpaca20k) |
|
|
|
prompt = "Question: [Your code-related question here] ### Answer: " |
|
inputs = tokenizer.encode(prompt, return_tensors="pt") |
|
outputs = model.generate(inputs) |
|
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
|
print(decoded_output) |
|
``` |
|
|
|
|
|
|
|
## License |
|
|
|
OpenRail |
|
|
|
--- |
|
|
|
|
