File size: 2,645 Bytes
0240a54 7562039 0240a54 7562039 0240a54 7562039 0240a54 d618791 0240a54 7562039 0240a54 2920594 0240a54 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 |
---
license: openrail
datasets:
- lucasmccabe-lmi/CodeAlpaca-20k
language:
- en
library_name: adapter-transformers
---
# Model Card for `opt350m-codealpaca20k`
## Model Description
An opt-350m model trained on the CodeAlpaca 20k dataset using quantization and Progressive Embedding Fine-Tuning (PEFT).
The resulting model is designed to understand and generate code-related responses based on the prompts provided.
[original model car](https://huggingface.co/facebook/opt-350m)
### Model Architecture
- **Base Model**: `facebook/opt-350m`
- **Fine-tuning**: Parameter-Efficient Fine-Tuning (PEFT)
## Training Data
The model was trained on the `lucasmccabe-lmi/CodeAlpaca-20k` dataset. This dataset contains code-related prompts and their corresponding outputs.
Script used for training is avaiable [here](https://github.com/harpomaxx/llm-finetuning/blob/0954a7ca16bb25bdef6ee9dd1089867bd4d8e0a5/code/python/scripts/stf_train_opt350m.py)
## Training Procedure
### Quantization Configuration:
- **Quantization Type**: 4-bit
- **Compute Dtype**: float16
- **Double Quant**: Enabled
### PEFT Configuration:
- **Lora Alpha**: 16
- **Lora Dropout**: 0.5
- **Bias**: None
- **Task Type**: CAUSAL_LM
- **Target Modules**: q_proj, v_proj, k_proj
### Training Arguments:
- **Batch Size**: 4 (per device)
- **Gradient Accumulation Steps**: 2
- **Number of Epochs**: 10
- **Optimizer**: `adamw_bnb_8bit`
- **Learning Rate**: 2e-5
- **Max Gradient Norm**: 0.3
- **Warmup Ratio**: 0.03
- **Learning Rate Scheduler**: Cosine
- **Logging Steps**: 10
- **Save Steps**: 250
- **FP16 Precision**: Enabled
### Training information from wandb
- **train/total_flos:** 72,761,086,854,758,400
- **train/train_loss:** 1.5557164267259171
- **train/train_runtime:** 5892.7285 seconds
- **train/train_steps_per_second:** 4.248
- **_runtime:** 5891.33976650238 seconds
- **_timestamp:** 1,695,390,058.0198596
- **train/epoch:** 10
- **train/global_step:** 25,030
- **train/learning_rate:** 8.371592860045851e-12
- **train/train_samples_per_second:** 33.977
- **_step:** 2,503
- **_wandb.runtime:** 5890 seconds
- **train/loss:** 1.4114
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/opt350m")
model = AutoModelForCausalLM.from_pretrained("harpomaxx/opt350m-codealpaca20k)
prompt = "Question: [Your code-related question here] ### Answer: "
inputs = tokenizer.encode(prompt, return_tensors="pt")
outputs = model.generate(inputs)
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(decoded_output)
```
## License
OpenRail
---
|