Model details

Quantized at 3.18bpw with hb 6. Can run full 4K context on 16GB VRAM. 8.13bpw also available.

Perplexity:

Base = 6.5820

8.13 = 6.5535

3.18 h6 = 6.6928

Dataset = wikitext

Prompt Format

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:

Inference Examples

Inference API (serverless) is not available, repository is disabled.