YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
Mostly untested!
RoPE Scaled QLoRA Long Context Extension of Llama-33b (LoRA)
Overview
This is base Llama-33b with minimal additional training to extend the useful context window.
- Context length extended to 16384 by RoPE Scaled Embeddings (Position Interpolation).
- Pretrained for additional 100 steps on 8192 length sequences from the pile dataset.
- The merged model is used as the starting point for training bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-LoRA
This is a QLoRA fine-tune
Pretraining took 10 hours on 1x RTX 6000 Ada.