Mistral 0.2 Reproduction, 32k context?
#1
by
saishf
- opened
This is a rebase merge using the formula from KatyTheCutie/LemonadeRP-4.5.3 on Mistral v0.2 7B base (instead of v0.1), for 32K context length (eliminating the 4K sliding window), with rope theta (re)set to 40K. No other changes were made.
- grimjim/lemonade-rebase-32k-7B
May be interesting?
@Nitral-AI 👀 new lemon juice?