Text Generation
Transformers
mergekit
Merge
alpaca
mistral
Not-For-All-Audiences
nsfw
Eval Results
Inference Endpoints
license: cc-by-nc-4.0 | |
library_name: transformers | |
tags: | |
- mergekit | |
- merge | |
- alpaca | |
- mistral | |
- not-for-all-audiences | |
- nsfw | |
model-index: | |
- name: IceCaffeLatteRP-7b | |
results: | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: AI2 Reasoning Challenge (25-Shot) | |
type: ai2_arc | |
config: ARC-Challenge | |
split: test | |
args: | |
num_few_shot: 25 | |
metrics: | |
- type: acc_norm | |
value: 69.03 | |
name: normalized accuracy | |
source: | |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=icefog72/IceCaffeLatteRP-7b | |
name: Open LLM Leaderboard | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: HellaSwag (10-Shot) | |
type: hellaswag | |
split: validation | |
args: | |
num_few_shot: 10 | |
metrics: | |
- type: acc_norm | |
value: 86.7 | |
name: normalized accuracy | |
source: | |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=icefog72/IceCaffeLatteRP-7b | |
name: Open LLM Leaderboard | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: MMLU (5-Shot) | |
type: cais/mmlu | |
config: all | |
split: test | |
args: | |
num_few_shot: 5 | |
metrics: | |
- type: acc | |
value: 64.34 | |
name: accuracy | |
source: | |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=icefog72/IceCaffeLatteRP-7b | |
name: Open LLM Leaderboard | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: TruthfulQA (0-shot) | |
type: truthful_qa | |
config: multiple_choice | |
split: validation | |
args: | |
num_few_shot: 0 | |
metrics: | |
- type: mc2 | |
value: 61.16 | |
source: | |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=icefog72/IceCaffeLatteRP-7b | |
name: Open LLM Leaderboard | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: Winogrande (5-shot) | |
type: winogrande | |
config: winogrande_xl | |
split: validation | |
args: | |
num_few_shot: 5 | |
metrics: | |
- type: acc | |
value: 81.29 | |
name: accuracy | |
source: | |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=icefog72/IceCaffeLatteRP-7b | |
name: Open LLM Leaderboard | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: GSM8k (5-shot) | |
type: gsm8k | |
config: main | |
split: test | |
args: | |
num_few_shot: 5 | |
metrics: | |
- type: acc | |
value: 64.06 | |
name: accuracy | |
source: | |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=icefog72/IceCaffeLatteRP-7b | |
name: Open LLM Leaderboard | |
quantized_by: bartowski | |
pipeline_tag: text-generation | |
## Exllama v2 Quantizations of IceCaffeLatteRP-7b | |
Using <a href="https://github.com/turboderp/exllamav2/releases/tag/v0.1.4">turboderp's ExLlamaV2 v0.1.4</a> for quantization. | |
<b>The "main" branch only contains the measurement.json, download one of the other branches for the model (see below)</b> | |
Each branch contains an individual bits per weight, with the main one containing only the meaurement.json for further conversions. | |
Original model: https://huggingface.co/icefog72/IceCaffeLatteRP-7b | |
## Prompt format | |
No chat template specified so default is used. This may be incorrect, check original model card for details. | |
``` | |
<s> [INST] <<SYS>> | |
{system_prompt} | |
<</SYS>> | |
{prompt} [/INST] </s> | |
``` | |
## Available sizes | |
| Branch | Bits | lm_head bits | VRAM (4k) | VRAM (16k) | VRAM (32k) | Description | | |
| ----- | ---- | ------- | ------ | ------ | ------ | ------------ | | |
| [8_0](https://huggingface.co/bartowski/IceCaffeLatteRP-7b-exl2/tree/8_0) | 8.0 | 8.0 | 8.4 GB | 9.8 GB | 11.8 GB | Maximum quality that ExLlamaV2 can produce, near unquantized performance. | | |
| [6_5](https://huggingface.co/bartowski/IceCaffeLatteRP-7b-exl2/tree/6_5) | 6.5 | 8.0 | 7.2 GB | 8.6 GB | 10.6 GB | Very similar to 8.0, good tradeoff of size vs performance, **recommended**. | | |
| [5_0](https://huggingface.co/bartowski/IceCaffeLatteRP-7b-exl2/tree/5_0) | 5.0 | 6.0 | 6.0 GB | 7.4 GB | 9.4 GB | Slightly lower quality vs 6.5, but usable on 8GB cards. | | |
| [4_25](https://huggingface.co/bartowski/IceCaffeLatteRP-7b-exl2/tree/4_25) | 4.25 | 6.0 | 5.3 GB | 6.7 GB | 8.7 GB | GPTQ equivalent bits per weight, slightly higher quality. | | |
| [3_5](https://huggingface.co/bartowski/IceCaffeLatteRP-7b-exl2/tree/3_5) | 3.5 | 6.0 | 4.7 GB | 6.1 GB | 8.1 GB | Lower quality, only use if you have to. | | |
## Download instructions | |
With git: | |
```shell | |
git clone --single-branch --branch 6_5 https://huggingface.co/bartowski/IceCaffeLatteRP-7b-exl2 IceCaffeLatteRP-7b-exl2-6_5 | |
``` | |
With huggingface hub (credit to TheBloke for instructions): | |
```shell | |
pip3 install huggingface-hub | |
``` | |
To download a specific branch, use the `--revision` parameter. For example, to download the 6.5 bpw branch: | |
Linux: | |
```shell | |
huggingface-cli download bartowski/IceCaffeLatteRP-7b-exl2 --revision 6_5 --local-dir IceCaffeLatteRP-7b-exl2-6_5 | |
``` | |
Windows (which apparently doesn't like _ in folders sometimes?): | |
```shell | |
huggingface-cli download bartowski/IceCaffeLatteRP-7b-exl2 --revision 6_5 --local-dir IceCaffeLatteRP-7b-exl2-6.5 | |
``` | |
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski |