nold commited on
Commit
b090d76
1 Parent(s): 50fe2e9

a518ff351e894ab6b16bc537b47af5f05a9403feef562458d8b87d59a866d2c1

Browse files
Files changed (1) hide show
  1. README.md +141 -0
README.md ADDED
@@ -0,0 +1,141 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: yi-license
4
+ license_link: https://huggingface.co/01-ai/Yi-34B/blob/main/LICENSE
5
+ language:
6
+ - en
7
+ library_name: transformers
8
+ base_model: []
9
+ tags:
10
+ - mergekit
11
+ - merge
12
+ - Yi
13
+ - exllama
14
+ - exllamav2
15
+ - exl2
16
+ ---
17
+ # RPMerge
18
+ A merge of several Yi 34B models with a singular goal: 40K+ context, instruct-enhanced storytelling.
19
+
20
+ Disappointed with some quirks of my previous kitchen sink merges (like token/instruct formats from various models showing up when they shouldn't), I've gone 'back to the basics' and picked a few Vicuna-format only models:
21
+
22
+ - [DrNicefellow/ChatAllInOne-Yi-34B-200K-V1](https://huggingface.co/DrNicefellow/ChatAllInOne-Yi-34B-200K-V1) and [migtissera/Tess-34B-v1.5b](https://huggingface.co/migtissera/Tess-34B-v1.5b) both have excellent general instruction-following performance.
23
+
24
+ - [cgato/Thespis-34b-v0.7](https://huggingface.co/cgato/Thespis-34b-v0.7) is trained on the "Username: {Input} / BotName: {Response}" format, to emphasize it in the merge (but not force it). It also seems to work for multi-character stories.
25
+
26
+ - [Doctor-Shotgun/limarpv3-yi-llama-34b-lora](https://huggingface.co/Doctor-Shotgun/limarpv3-yi-llama-34b-lora) is trained on roleplaying data, but merged at a modest weight to not over emphasize it. This is the only non-vicuna model (being alpaca format), but it doesn't seem to interefere with the Vicuna format or adversely affect long-context perplexity
27
+
28
+ - [adamo1139/yi-34b-200k-rawrr-dpo-2](https://huggingface.co/adamo1139/yi-34b-200k-rawrr-dpo-2) the base for the limarp lora, this is base Yi gently finetuned to discourage refusals.
29
+
30
+ - [migtissera/Tess-M-Creative-v1.0](https://huggingface.co/migtissera/Tess-M-Creative-v1.0) and [NousResearch/Nous-Capybara-34B](https://huggingface.co/NousResearch/Nous-Capybara-34B) are both "undertrained" Yi models. I find they excel at raw completion performance (like long novel continuations) while still retaining some Vicuna instruct ability. This may be why some still prefer the original Tess 1.0/Capybara merge.
31
+
32
+ I consider this a more "focused" merge that previous ones. I will investigate other models (perhaps chatML models?) for a more "factual assistant" focused merge, as well as a coding-focused merge if I can't find one to suit my needs.
33
+
34
+
35
+ ## Prompt template: Orca-Vicuna
36
+ ```
37
+ SYSTEM: {system_message}
38
+ USER: {prompt}
39
+ ASSISTANT:
40
+ ```
41
+ Raw prompting as described here is also effective: https://old.reddit.com/r/LocalLLaMA/comments/18zqy4s/the_secret_to_writing_quality_stories_with_llms/
42
+
43
+ As well as a very explicit system prompt like this: https://old.reddit.com/r/LocalLLaMA/comments/1aiz6zu/roleplaying_system_prompts/koygiwa/
44
+
45
+
46
+ ## Running
47
+
48
+ Chinese models with large tokenizer vocabularies like Yi need *careful* parameter tuning due to their huge logit sampling "tails." Yi in particular also runs relatively "hot" even at lower temperatures.
49
+
50
+ I am a huge fan of Kalomaze's quadratic sampling (shown as "smoothing factor" where available), as described here: https://github.com/oobabooga/text-generation-webui/pull/5403
51
+
52
+ Otherwise, I recommend a lower temperature with 0.1 or higher MinP, a little repetition penalty, and mirostat with a low tau, and no other samplers. See the explanation here: https://github.com/ggerganov/llama.cpp/pull/3841
53
+
54
+ 24GB GPUs can efficiently run Yi-34B-200K models at **40K-90K context** with exllamav2, and performant UIs like [exui](https://github.com/turboderp/exui). I go into more detail in this [post](https://old.reddit.com/r/LocalLLaMA/comments/1896igc/how_i_run_34b_models_at_75k_context_on_24gb_fast/). Empty 16GB GPUs can still run the high context with aggressive quantization.
55
+
56
+ To load/train this in full-context backends like transformers, you *must* change `max_position_embeddings` in config.json to a lower value than 200,000, otherwise you will OOM! I do not recommend running high context without context-efficient backends that support flash attention + 8 bit kv cache, like exllamav2, litellm, vllm or unsloth.
57
+
58
+
59
+ ## Testing Notes
60
+
61
+ Thanks to ParasiticRogue for this idea of a Vicuna-only merge, see: https://huggingface.co/brucethemoose/jondurbin_bagel-dpo-34b-v0.2-exl2-4bpw-fiction/discussions
62
+
63
+ See: https://huggingface.co/brucethemoose/Yi-34B-200K-DARE-megamerge-v8#testing-notes
64
+
65
+ This is a possible base for a storytelling finetune/LASER in the future, once I can bite the bullet and rent some A100s or a MI300.
66
+
67
+ I have tested this merge with with novel-style continuation (but not much chat-style roleplay), and some assistant-style responses and long context analysis. I haven't seen any refusals so far.
68
+
69
+ ## Merge Details
70
+ ### Merge Method
71
+
72
+ This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method using /home/alpha/Models/Raw/chargoddard_Yi-34B-200K-Llama as a base.
73
+
74
+ ### Models Merged
75
+
76
+ The following models were included in the merge:
77
+ * /home/alpha/Models/Raw/migtissera_Tess-34B-v1.5b
78
+ * /home/alpha/Models/Raw/migtissera_Tess-M-Creative-v1.0
79
+ * /home/alpha/Models/Raw/cgato_Thespis-34b-DPO-v0.7
80
+ * /home/alpha/Models/Raw/Nous-Capybara-34B
81
+ * /home/alpha/Models/Raw/admo_limarp
82
+ * /home/alpha/Models/Raw/DrNicefellow_ChatAllInOne-Yi-34B-200K-V1
83
+
84
+ ### Configuration
85
+
86
+ The following YAML configuration was used to produce this model:
87
+
88
+ ```yaml
89
+ models:
90
+ - model: /home/alpha/Models/Raw/chargoddard_Yi-34B-200K-Llama
91
+ # No parameters necessary for base model
92
+ - model: /home/alpha/Models/Raw/migtissera_Tess-34B-v1.5b
93
+ #Emphasize the beginning of Vicuna format models
94
+ parameters:
95
+ weight: 0.19
96
+ density: 0.59
97
+ - model: /home/alpha/Models/Raw/Nous-Capybara-34B
98
+ parameters:
99
+ weight: 0.19
100
+ density: 0.55
101
+ # Vicuna format
102
+ - model: /home/alpha/Models/Raw/migtissera_Tess-M-Creative-v1.0
103
+ parameters:
104
+ weight: 0.05
105
+ density: 0.55
106
+ - model: /home/alpha/Models/Raw/DrNicefellow_ChatAllInOne-Yi-34B-200K-V1
107
+ parameters:
108
+ weight: 0.19
109
+ density: 0.55
110
+ - model: adamo1139/yi-34b-200k-rawrr-dpo-2+Doctor-Shotgun/limarpv3-yi-llama-34b-lora
111
+ parameters:
112
+ weight: 0.19
113
+ density: 0.48
114
+ - model: /home/alpha/Models/Raw/cgato_Thespis-34b-DPO-v0.7
115
+ parameters:
116
+ weight: 0.19
117
+ density: 0.59
118
+
119
+
120
+ merge_method: dare_ties
121
+ tokenizer_source: union
122
+ base_model: /home/alpha/Models/Raw/chargoddard_Yi-34B-200K-Llama
123
+ parameters:
124
+ int8_mask: true
125
+ dtype: bfloat16
126
+ ```
127
+
128
+
129
+ ## Self Promotion
130
+
131
+ I'm part of a AI startup called Holocene AI!
132
+
133
+ We're new, busy, and still setting things up. But if you have any business inquiries, want a job, or just want some consultation, feel free to shoot me an email. We have expertise in RAG applications and llama/embeddings model finetuning, and absolutely *none* of the nonsense of scammy AI startups.
134
+
135
+ Contact me at: agates.holocene.ai@gmail.com
136
+
137
+ I also set up a Ko-Fi! I want to run some (personal) training/LASERing as well, at 100K context or so. If you'd like to buy me 10 minutes on an A100 (or 5 seconds on an MI300X), I'd appreciate it: https://ko-fi.com/alphaatlas
138
+
139
+ ***
140
+
141
+ Vanilla Quantization by [nold](https://huggingface.co/nold), Original Model [brucethemoose/Yi-34B-200K-RPMerge](https://huggingface.co/brucethemoose/Yi-34B-200K-RPMerge). Created using [llm-quantizer](https://github.com/Nold360/llm-quantizer) Pipeline - 0e95dcd401087b713c2eca7c89ff8108e61969f0