Create README.md

README.md

@@ -0,0 +1,83 @@
+---
+license: openrail
+datasets:
+- lucasmccabe-lmi/CodeAlpaca-20k
+language:
+- en
+library_name: adapter-transformers
+---
+
+# Model Card for `opt350m-codealpaca20k`
+
+## Model Description
+
+A simple opt350m model  trained on the CodeAlpaca dataset using quantization and Progressive Embedding Fine-Tuning (PEFT). It's designed to understand and generate code-related responses based on the prompts provided.
+
+### Model Architecture
+
+- **Base Model**: `facebook/opt-350m`
+- **Fine-tuning**: Progressive Embedding Fine-Tuning (PEFT)
+
+## Training Data
+
+The model was trained on the `lucasmccabe-lmi/CodeAlpaca-20k` dataset. This dataset contains code-related prompts and their corresponding outputs.
+
+## Training Procedure
+
+### Quantization Configuration:
+
+- **Quantization Type**: 4-bit
+- **Compute Dtype**: float16
+- **Double Quant**: Enabled
+
+### PEFT Configuration:
+
+- **Lora Alpha**: 16
+- **Lora Dropout**: 0.5
+- **Bias**: None
+- **Task Type**: CAUSAL_LM
+- **Target Modules**: q_proj, v_proj, k_proj
+
+### Training Arguments:
+
+- **Output Directory**: `./results`
+- **Batch Size**: 4 (per device)
+- **Gradient Accumulation Steps**: 2
+- **Number of Epochs**: 10
+- **Optimizer**: `adamw_bnb_8bit`
+- **Learning Rate**: 2e-5
+- **Max Gradient Norm**: 0.3
+- **Warmup Ratio**: 0.03
+- **Learning Rate Scheduler**: Cosine
+- **Logging Steps**: 10
+- **Save Steps**: 250
+- **FP16 Precision**: Enabled
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("facebook/opt350m")
+model = AutoModelForCausalLM.from_pretrained("harpomaxx/opt350m-codealpaca20k)
+
+prompt = "### Question: [Your code-related question here]"
+inputs = tokenizer.encode(prompt, return_tensors="pt")
+outputs = model.generate(inputs)
+decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+print(decoded_output)
+```
+
+## Limitations and Bias
+
+- The model is specifically fine-tuned for code-related tasks, and its performance on other tasks might not be optimal.
+- The biases in the CodeAlpaca dataset might be reflected in the model's outputs.
+
+## License
+
+[Specify the license under which the model is released.]
+
+---
+
+Remember to replace placeholders like `your_model_name_here` with the actual name or path of your model. Adjust any other details as necessary to fit the specifics of your model and its training.