MMLU is 25?????
#3
by
sleepyjoecheated
- opened
What?
Context: questions in MMLU have four choices each, so MMLU=25 means the model is not performing any better than random chance.
Yup, Craig Federighi went all in on prioritizing his skydiving and parkour stunts at WWDC 2024 instead of leveraging more of Apple’s massive $3.3 trillion to leverage more privacy-preserving on-device differential privacy and federated learning for better language models.