---
base_model:
- yuvraj17/EvolCodeLlama-3.1-8B-Instruct
- yzhuang/Meta-Llama-3-8B-Instruct_fictional_gsm8k_English_v1
tags:
- merge
- mergekit
- lazymergekit
- yuvraj17/EvolCodeLlama-3.1-8B-Instruct
- yzhuang/Meta-Llama-3-8B-Instruct_fictional_gsm8k_English_v1
---

# Llama3-8B-Instruct-Slerp

Llama3-8B-Instruct-Slerp is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
* [yuvraj17/EvolCodeLlama-3.1-8B-Instruct](https://huggingface.co/yuvraj17/EvolCodeLlama-3.1-8B-Instruct)
* [yzhuang/Meta-Llama-3-8B-Instruct_fictional_gsm8k_English_v1](https://huggingface.co/yzhuang/Meta-Llama-3-8B-Instruct_fictional_gsm8k_English_v1)

## 🧩 Configuration

```yaml
slices:
  - sources:
      - model: yuvraj17/EvolCodeLlama-3.1-8B-Instruct
        layer_range: [0, 32]
      - model: yzhuang/Meta-Llama-3-8B-Instruct_fictional_gsm8k_English_v1
        layer_range: [0, 32]
merge_method: slerp
base_model: yuvraj17/EvolCodeLlama-3.1-8B-Instruct
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: float16

```

## 💻 Usage

```python
!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "yuvraj17/Llama3-8B-Instruct-Slerp"
messages = [{"role": "user", "content": "What is a large language model?"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```

# 🏆 Evaluation Scores

## Nous

|                                                     Model                                   |AGIEval|TruthfulQA|Bigbench|
|---------------------------------------------------------------------------------------------|------:|---------:|-------:|
|[yuvraj17/Llama3-8B-Instruct-Slerp](https://huggingface.co/yuvraj17/Llama3-8B-Instruct-Slerp)|  33.28|     49.78|   35.38|


### AGIEval
|             Task             |Version| Metric  | Value |   | Stderr |
|------------------------------|------:|---------|------:|---|-------:|
| agieval_aqua_rat              |      0| acc     | 23.62 |±  |  2.67  |
|                              |       | acc_norm| 22.05 |±  |  2.61  |
| agieval_logiqa_en             |      0| acc     | 27.50 |±  |  1.75  |
|                              |       | acc_norm| 31.80 |±  |  1.83  |
| agieval_lsat_ar               |      0| acc     | 21.30 |±  |  2.71  |
|                              |       | acc_norm| 20.87 |±  |  2.69  |
| agieval_lsat_lr               |      0| acc     | 35.29 |±  |  2.12  |
|                              |       | acc_norm| 37.65 |±  |  2.15  |
| agieval_lsat_rc               |      0| acc     | 42.01 |±  |  3.01  |
|                              |       | acc_norm| 39.78 |±  |  2.99  |
| agieval_sat_en                |      0| acc     | 55.83 |±  |  3.47  |
|                              |       | acc_norm| 50.49 |±  |  3.49  |
| agieval_sat_en_without_passage|      0| acc     | 36.89 |±  |  3.37  |
|                              |       | acc_norm| 34.95 |±  |  3.33  |
| agieval_sat_math              |      0| acc     | 29.55 |±  |  3.08  |
|                              |       | acc_norm| 28.64 |±  |  3.05  |

**Average score**: 33.28%

### TruthfulQA


|        Task         |Version| Metric | Value |   | Stderr |
|---------------------|------:|--------|------:|---|-------:|
| truthfulqa_mc       |      1| mc1    | 33.54 |±  |  1.65  |
|                     |       | mc2    | 49.78 |±  |  1.53  |

**Average score**: 49.78%

### BigBench

|                Task                |Version|        Metric         | Value |   | Stderr |
|------------------------------------|------:|-----------------------|------:|---|-------:|
| bigbench_causal_judgement          |      0| multiple_choice_grade  | 47.89 |±  |  3.63  |
| bigbench_date_understanding        |      0| multiple_choice_grade  | 39.02 |±  |  2.54  |
| bigbench_disambiguation_qa         |      0| multiple_choice_grade  | 33.72 |±  |  2.95  |
| bigbench_geometric_shapes          |      0| multiple_choice_grade  | 20.61 |±  |  2.14  |
| bigbench_logical_deduction_five_objects|  0| multiple_choice_grade  | 31.40 |±  |  2.08  |
| bigbench_logical_deduction_seven_objects| 0| multiple_choice_grade  | 23.71 |±  |  1.61  |
| bigbench_logical_deduction_three_objects| 0| multiple_choice_grade  | 47.00 |±  |  2.89  |
| bigbench_movie_recommendation      |      0| multiple_choice_grade  | 27.40 |±  |  1.99  |
| bigbench_navigate                  |      0| multiple_choice_grade  | 50.10 |±  |  1.58  |
| bigbench_reasoning_about_colored_objects| 0| multiple_choice_grade  | 38.40 |±  |  1.09  |
| bigbench_ruin_names                |      0| multiple_choice_grade  | 27.23 |±  |  2.11  |
| bigbench_salient_translation_error_detection| 0| multiple_choice_grade  | 25.45 |±  |  1.38  |
| bigbench_snarks                    |      0| multiple_choice_grade  | 46.41 |±  |  3.72  |
| bigbench_sports_understanding      |      0| multiple_choice_grade  | 50.30 |±  |  1.59  |
| bigbench_temporal_sequences        |      0| multiple_choice_grade  | 37.30 |±  |  1.53  |
| bigbench_tracking_shuffled_objects_five_objects| 0| multiple_choice_grade  | 21.36 |±  |  1.16  |
| bigbench_tracking_shuffled_objects_seven_objects| 0| multiple_choice_grade  | 17.14 |±  |  0.90  |
| bigbench_tracking_shuffled_objects_three_objects| 0| multiple_choice_grade  | 47.00 |±  |  2.89  |

**Average score**: 35.38%