dnhkng commited on
Commit
3707c0e
1 Parent(s): a28de8f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -3
README.md CHANGED
@@ -97,10 +97,21 @@ model-index:
97
  name: Open LLM Leaderboard
98
  ---
99
 
100
- This is a new kind of model optimization.
101
- This model is based on MaziyarPanahi/calme-2.1-qwen2-72b, which was tuned from Qwen2-72B.
102
 
103
- A paper on the technique is currently being written. Special thanks to my wife, for putting up with me coding in the basement for too many evenings and weekends for months!
 
 
 
 
 
 
 
 
 
 
 
 
104
 
105
  This research was supported with hardware from the [appliedAI Institute](https://www.appliedai-institute.de/en/), whose goal is to generate and communicate high-quality knowledge about trustworthy AI.
106
 
 
97
  name: Open LLM Leaderboard
98
  ---
99
 
100
+ This is a new kind of model optimization. It is based on a new method for the analysis of the functional role of layers within the transformer stack, and on layer duplication (self-merging) to increase intelligence.
 
101
 
102
+ *No Weights were modified in this process!*
103
+
104
+ ### Model improvement (%) with layer duplication:
105
+ | | Average | IFEval | BBH | MATH Lvl 5 | GPQA | MUSR | MMLU-PRO |
106
+ |-----------------|---------|--------|------|------------|------|-------|----------|
107
+ | RYS Improvement | 2.61 | -2.05 | 2.51 | 8.16 | 2.58 | 17.72 | 0.31 |
108
+
109
+
110
+ This model is based on MaziyarPanahi/calme-2.1-qwen2-72b, which was tuned from Qwen2-72B. As this method is orthogonal to fine-tuning, the further finetune from MaziyarPanahi now has the top position:
111
+ https://huggingface.co/MaziyarPanahi/calme-2.4-rys-78b
112
+
113
+
114
+ A paper on the technique is currently being written. Currently, all four top models on the leaderboard are based on the RYS method. Special thanks to my wife, for putting up with me coding in the basement for too many evenings and weekends for months!
115
 
116
  This research was supported with hardware from the [appliedAI Institute](https://www.appliedai-institute.de/en/), whose goal is to generate and communicate high-quality knowledge about trustworthy AI.
117