File size: 1,884 Bytes
0240a54 2920594 0240a54 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 |
---
license: openrail
datasets:
- lucasmccabe-lmi/CodeAlpaca-20k
language:
- en
library_name: adapter-transformers
---
# Model Card for `opt350m-codealpaca20k`
## Model Description
A simple opt350m model trained on the CodeAlpaca dataset using quantization and Progressive Embedding Fine-Tuning (PEFT). It's designed to understand and generate code-related responses based on the prompts provided.
### Model Architecture
- **Base Model**: `facebook/opt-350m`
- **Fine-tuning**: Progressive Embedding Fine-Tuning (PEFT)
## Training Data
The model was trained on the `lucasmccabe-lmi/CodeAlpaca-20k` dataset. This dataset contains code-related prompts and their corresponding outputs.
## Training Procedure
### Quantization Configuration:
- **Quantization Type**: 4-bit
- **Compute Dtype**: float16
- **Double Quant**: Enabled
### PEFT Configuration:
- **Lora Alpha**: 16
- **Lora Dropout**: 0.5
- **Bias**: None
- **Task Type**: CAUSAL_LM
- **Target Modules**: q_proj, v_proj, k_proj
### Training Arguments:
- **Output Directory**: `./results`
- **Batch Size**: 4 (per device)
- **Gradient Accumulation Steps**: 2
- **Number of Epochs**: 10
- **Optimizer**: `adamw_bnb_8bit`
- **Learning Rate**: 2e-5
- **Max Gradient Norm**: 0.3
- **Warmup Ratio**: 0.03
- **Learning Rate Scheduler**: Cosine
- **Logging Steps**: 10
- **Save Steps**: 250
- **FP16 Precision**: Enabled
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/opt350m")
model = AutoModelForCausalLM.from_pretrained("harpomaxx/opt350m-codealpaca20k)
prompt = "### Question: [Your code-related question here]"
inputs = tokenizer.encode(prompt, return_tensors="pt")
outputs = model.generate(inputs)
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(decoded_output)
```
## License
OpenRail
---
|