Model Card

We provide a low-rank adapter for an instruction-tuned 12B-parameter GPT3-style language model.

Prompting

For instructions that do not require extraneous inputs, the recommended prompt is:

Below is an instruction that describes a task. Write a response that appropriately completes the request.


### Instruction:

< -- instruction goes here --- >

### Response:

For instructions that do require extraneous inputs, the recommended prompt is:

Below is an instruction that describes a task. Write a response that appropriately completes the request.


### Instruction:

< -- instruction goes here -- >

### Input:

< -- extraneous input goes here -- >

### Response:

Since the model performs causal language modeling, the model's response to the prompt is the text completing the sequence beginning with the prompt.

Instruction-Tuning

This model was instruction-tuned on an internally-curated Alpaca-style dataset.

Epochs: 3
Batch size: 128
Cutoff length: 2048
Learning rate: 3.2e-5, cosine decay
LoRA r: 8
LoRA alpha: 16
LoRA dropout: 0.0
LoRA target modules: 'query_key_value', 'dense_h_to_4h', 'dense_4h_to_h', 'dense'
License: The instruction-tuning data is subject to the Creative Commons 4.0 license.

Base Model

This model was instruction-tuned from a 12B variant from the Pythia family.

Repository: EleutherAI/pythia-12b
Paper: arxiv:2304.01373
License: The base model is subject to the Apache 2.0 license.
Model type: Transformer-based Language Model

Licensing Information

We release this adapter under the Creative Commons NonCommercial (CC BY-NC 4.0) license.