---
license: openrail
datasets:
- lucasmccabe-lmi/CodeAlpaca-20k
language:
- en
library_name: adapter-transformers
---

# Model Card for `opt350m-codealpaca20k`

## Model Description

An  opt-350m model  trained on the CodeAlpaca  20k dataset using quantization and Progressive Embedding Fine-Tuning (PEFT). 
The resulting model is  designed to understand and generate code-related responses based on the prompts provided.

[original model car](https://huggingface.co/facebook/opt-350m)

### Model Architecture

- **Base Model**: `facebook/opt-350m`
- **Fine-tuning**: Parameter-Efficient Fine-Tuning (PEFT)

## Training Data

The model was trained on the `lucasmccabe-lmi/CodeAlpaca-20k` dataset. This dataset contains code-related prompts and their corresponding outputs.
Script used for training is avaiable [here](https://github.com/harpomaxx/llm-finetuning/blob/0954a7ca16bb25bdef6ee9dd1089867bd4d8e0a5/code/python/scripts/stf_train_opt350m.py)

## Training Procedure

### Quantization Configuration:

- **Quantization Type**: 4-bit
- **Compute Dtype**: float16
- **Double Quant**: Enabled

### PEFT Configuration:

- **Lora Alpha**: 16
- **Lora Dropout**: 0.5
- **Bias**: None
- **Task Type**: CAUSAL_LM
- **Target Modules**: q_proj, v_proj, k_proj

### Training Arguments:

- **Batch Size**: 4 (per device)
- **Gradient Accumulation Steps**: 2
- **Number of Epochs**: 10
- **Optimizer**: `adamw_bnb_8bit`
- **Learning Rate**: 2e-5
- **Max Gradient Norm**: 0.3
- **Warmup Ratio**: 0.03
- **Learning Rate Scheduler**: Cosine
- **Logging Steps**: 10
- **Save Steps**: 250
- **FP16 Precision**: Enabled


### Training information from wandb

- **train/total_flos:** 72,761,086,854,758,400
- **train/train_loss:** 1.5557164267259171
- **train/train_runtime:** 5892.7285 seconds
- **train/train_steps_per_second:** 4.248
- **_runtime:** 5891.33976650238 seconds
- **_timestamp:** 1,695,390,058.0198596
- **train/epoch:** 10
- **train/global_step:** 25,030
- **train/learning_rate:** 8.371592860045851e-12
- **train/train_samples_per_second:** 33.977
- **_step:** 2,503
- **_wandb.runtime:** 5890 seconds
- **train/loss:** 1.4114






## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt350m")
model = AutoModelForCausalLM.from_pretrained("harpomaxx/opt350m-codealpaca20k)

prompt = "Question: [Your code-related question here] ### Answer: "
inputs = tokenizer.encode(prompt, return_tensors="pt")
outputs = model.generate(inputs)
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(decoded_output)
```



## License

OpenRail

---