Now this is what I like to see
8x120b MoE next? no balls?
anyways good job.
Just a question, why use Euryale-1.3-L2-70B rather than something with better benchmark scores like MoMo-72B, qwen or Aurora-Nights-70B?
Benchmarks are a meme
Benchmarks are a meme
I mean yeah you are right, but this model used is months old
Euryale + Xwin is just a known working formula that was used for goliath, one of the best local models ever made. In fact, it's so good, that even my attempt at recreating it by combining WinterGoddess(replacement for Euryale) and Xwin failed(failed not as in it's dumb, but in the sense that it just isn't what I want from RP model, too much of positive assistant is seeping through). So far I could not find a suitable replacement for Euryale, the closest thing that I found is Dolphin, but it has it's own quirks. Aurora nights is not a bad suggestion, but it is just a bit too aligned to be be perfect. If you want to see a list of attempts of other people at recreating goliath, you can check out this collection. It has a lot of models you probably have never heard of.
I also have my own mememark, it evalues for stuff which automated benchmark doesn't.
Interesting.
I was unaware goliath used euryale lol, I thought it was just a feature of this LLM alone. I will check out the collection.
For potential over-alignment you could run it over the toxic dpo or over bagel for finetuning potentially.
From what I've heard, DPO has not yet been perfected. Maybe in the future it will be possible.
Yeah, I see that as well, it seems to reduce the dynamic nature of models. That being said, the toxic dpo is quite small so it likely won't have much of a negative effect.
benchmarks are a meme
Well, so is this model. I never expected it to be smart. Maybe I'll recreate this properly if miqudev releases fp16 weights, or if I figure out proper weight dequantization.
@alpindale Try self-merging miqu. I had success with self-merging models in the past.
also maybe remake with miqu sf rather than fp16 repo