---
license: openrail
datasets:
- lucasmccabe-lmi/CodeAlpaca-20k
language:
- en
library_name: adapter-transformers
---

# Model Card for `opt350m-codealpaca20k`

## Model Description

A simple opt350m model  trained on the CodeAlpaca dataset using quantization and Progressive Embedding Fine-Tuning (PEFT). It's designed to understand and generate code-related responses based on the prompts provided.

### Model Architecture

- **Base Model**: `facebook/opt-350m`
- **Fine-tuning**: Progressive Embedding Fine-Tuning (PEFT)

## Training Data

The model was trained on the `lucasmccabe-lmi/CodeAlpaca-20k` dataset. This dataset contains code-related prompts and their corresponding outputs.

## Training Procedure

### Quantization Configuration:

- **Quantization Type**: 4-bit
- **Compute Dtype**: float16
- **Double Quant**: Enabled

### PEFT Configuration:

- **Lora Alpha**: 16
- **Lora Dropout**: 0.5
- **Bias**: None
- **Task Type**: CAUSAL_LM
- **Target Modules**: q_proj, v_proj, k_proj

### Training Arguments:

- **Output Directory**: `./results`
- **Batch Size**: 4 (per device)
- **Gradient Accumulation Steps**: 2
- **Number of Epochs**: 10
- **Optimizer**: `adamw_bnb_8bit`
- **Learning Rate**: 2e-5
- **Max Gradient Norm**: 0.3
- **Warmup Ratio**: 0.03
- **Learning Rate Scheduler**: Cosine
- **Logging Steps**: 10
- **Save Steps**: 250
- **FP16 Precision**: Enabled

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt350m")
model = AutoModelForCausalLM.from_pretrained("harpomaxx/opt350m-codealpaca20k)

prompt = "### Question: [Your code-related question here]"
inputs = tokenizer.encode(prompt, return_tensors="pt")
outputs = model.generate(inputs)
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(decoded_output)
```



## License

OpenRail

---