|
--- |
|
license: apache-2.0 |
|
tags: |
|
- moe |
|
- frankenmoe |
|
- merge |
|
- mergekit |
|
- lazymergekit |
|
- aloobun/Cypher-Mini-Laser-1.8B |
|
- aloobun/Cypher-CoT-Laser-1.8B |
|
- laser |
|
base_model: |
|
- aloobun/Cypher-Mini-Laser-1.8B |
|
- aloobun/Cypher-CoT-Laser-1.8B |
|
datasets: |
|
- kaist-ai/CoT-Collection |
|
- Locutusque/Hercules-v3.0 |
|
--- |
|
|
|
This work is based on implementation of the LASER technique, by @fernandofernandes & @ehartford. |
|
|
|
I 'lasered' two models first and then merged them. |
|
|
|
# Cypher-Laser-Mixtral-2x1.8B-v0.1 |
|
|
|
Cypher-Laser-Mixtral-2x1.8B-v0.1 is a Mixure of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing): |
|
* [aloobun/Cypher-Mini-Laser-1.8B](https://huggingface.co/aloobun/Cypher-Mini-Laser-1.8B) |
|
* [aloobun/Cypher-CoT-Laser-1.8B](https://huggingface.co/aloobun/Cypher-CoT-Laser-1.8B) |
|
|
|
## 🧩 Configuration |
|
|
|
```yaml |
|
base_model: aloobun/Cypher-Mini-Laser-1.8B |
|
gate_mode: hidden |
|
dtype: bfloat16 |
|
experts: |
|
- source_model: aloobun/Cypher-Mini-Laser-1.8B |
|
positive_prompts: |
|
- "Write a Python script that sorts a list of integers using the bubble sort algorithm." |
|
- "Write a JavaScript function that redirects a web page to another page after 5 seconds." |
|
- "Describe the steps to troubleshoot a fluid dynamics issue with a water fountain." |
|
- "Write a short story about a knight's quest to find a lost treasure, and then summarize it in one paragraph." |
|
- "Summarize the following article with details and clarity." |
|
- "Tell me about your favorite book and why you like it." |
|
|
|
- source_model: aloobun/Cypher-CoT-Laser-1.8B |
|
positive_prompts: |
|
- "Liam saw an animal running on the farm. Q: Is it true that The animal could be a horse." |
|
- "Based on the following paragraph can we conclude that the sentence below is true?" |
|
- "According to the article, how do dolphins communicate with each other?" |
|
- "Solve this math problem Solve 7644 = 4648*d - 4557*d for d." |
|
- "If we have 3 marbles, and two roll under the counter, and one is found, how many marbles are there?" |
|
- "What is the result of 25 divided by 5?" |
|
- "Is it morally justifiable to lie to protect someone's feelings?" |
|
- "Determine if the sentence is true based on the text below. Choose from options." |
|
- "What might a person do if they forget their umbrella on a rainy day?" |
|
- "Which of the following is an example of renewable energy: a) Coal, b) Solar, c) Oil, d) Natural gas?" |
|
- "What is the capital of Canada? a) Toronto, b) Ottawa, c) Montreal, d) Vancouver." |
|
- "Which of these animals is a mammal? a) Snake, b) Dolphin, c) Turtle, d) Frog." |
|
- "Given a story, answer the question about the story." |
|
- "Given a prompt and four completions, select the completion that is the most plausible in continuing or answering the prompt." |
|
``` |
|
|
|
|
|
## Format: |
|
``` |
|
<|system|></s><|prompt|></s><|answer|> |
|
``` |
