harpomaxx's picture
Update README.md
d618791
metadata
license: openrail
datasets:
  - lucasmccabe-lmi/CodeAlpaca-20k
language:
  - en
library_name: adapter-transformers

Model Card for opt350m-codealpaca20k

Model Description

An opt-350m model trained on the CodeAlpaca 20k dataset using quantization and Progressive Embedding Fine-Tuning (PEFT). The resulting model is designed to understand and generate code-related responses based on the prompts provided.

original model car

Model Architecture

  • Base Model: facebook/opt-350m
  • Fine-tuning: Parameter-Efficient Fine-Tuning (PEFT)

Training Data

The model was trained on the lucasmccabe-lmi/CodeAlpaca-20k dataset. This dataset contains code-related prompts and their corresponding outputs. Script used for training is avaiable here

Training Procedure

Quantization Configuration:

  • Quantization Type: 4-bit
  • Compute Dtype: float16
  • Double Quant: Enabled

PEFT Configuration:

  • Lora Alpha: 16
  • Lora Dropout: 0.5
  • Bias: None
  • Task Type: CAUSAL_LM
  • Target Modules: q_proj, v_proj, k_proj

Training Arguments:

  • Batch Size: 4 (per device)
  • Gradient Accumulation Steps: 2
  • Number of Epochs: 10
  • Optimizer: adamw_bnb_8bit
  • Learning Rate: 2e-5
  • Max Gradient Norm: 0.3
  • Warmup Ratio: 0.03
  • Learning Rate Scheduler: Cosine
  • Logging Steps: 10
  • Save Steps: 250
  • FP16 Precision: Enabled

Training information from wandb

  • train/total_flos: 72,761,086,854,758,400
  • train/train_loss: 1.5557164267259171
  • train/train_runtime: 5892.7285 seconds
  • train/train_steps_per_second: 4.248
  • _runtime: 5891.33976650238 seconds
  • _timestamp: 1,695,390,058.0198596
  • train/epoch: 10
  • train/global_step: 25,030
  • train/learning_rate: 8.371592860045851e-12
  • train/train_samples_per_second: 33.977
  • _step: 2,503
  • _wandb.runtime: 5890 seconds
  • train/loss: 1.4114

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt350m")
model = AutoModelForCausalLM.from_pretrained("harpomaxx/opt350m-codealpaca20k)

prompt = "Question: [Your code-related question here] ### Answer: "
inputs = tokenizer.encode(prompt, return_tensors="pt")
outputs = model.generate(inputs)
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(decoded_output)

License

OpenRail