Every time a new model is released that is topping 10+ leaderboards on 50+ benchmarks... 🚀
text
Every time a new model is released that is topping 10+ leaderboards on 50+ benchmarks... 🚀
new_line
new_line
My brain goes... I will wait for the LMSYS Chatbot Arena results! 🤔
text
My brain goes... I will wait for the LMSYS Chatbot Arena results! 🤔
new_line
new_line
User-facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. 🐢
text
User-facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. 🐢
new_line
new_line
Now we have MixEval, a new open benchmark with a 96% correlation to LMSYS Chatbot Arena and Human preferences. 🎯
text
Now we have MixEval, a new open benchmark with a 96% correlation to LMSYS Chatbot Arena and Human preferences. 🎯
new_line
new_line
It comes with MixEval (4k samples) and MixEval Hard (1k samples) 📊
text
It comes with MixEval (4k samples) and MixEval Hard (1k samples) 📊
new_line
new_line
Can use GPT-3.5-Turbo or any other open-source models as Parser/Judge 🤖
text
Can use GPT-3.5-Turbo or any other open-source models as Parser/Judge 🤖
new_line
new_line
It takes less than 6% of the time and cost of MMLU 💸
text
It takes less than 6% of the time and cost of MMLU 💸
new_line
new_line
As expected:
text
As expected:
new_line
In open models: Qwen2 72B >> Llama 3 70B >> Mixtral 8x7B 🔝
text
In open models: Qwen2 72B >> Llama 3 70B >> Mixtral 8x7B 🔝
new_line
In Closed Models: GPT-4o >> Claude 3 Opus >> Gemini Pro 🔒
text
In Closed Models: GPT-4o >> Claude 3 Opus >> Gemini Pro 🔒
new_line
new_line
Leaderboard: 
text
Leaderboard: 
https://mixeval.github.io/
https://mixeval.github.io/
link
  📈
text
  📈
Every time a new model is released that is topping 10+ leaderboards on 50+ benchmarks... 🚀

My brain goes... I will wait for the LMSYS Chatbot Arena results! 🤔

User-facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. 🐢

Now we have MixEval, a new open benchmark with a 96% correlation to LMSYS Chatbot Arena and Human preferences. 🎯

It comes with MixEval (4k samples) and MixEval Hard (1k samples) 📊

Can use GPT-3.5-Turbo or any other open-source models as Parser/Judge 🤖

It takes less than 6% of the time and cost of MMLU 💸

As expected:
In open models: Qwen2 72B >> Llama 3 70B >> Mixtral 8x7B 🔝
In Closed Models: GPT-4o >> Claude 3 Opus >> Gemini Pro 🔒

Leaderboard: https://mixeval.github.io/  📈
👀
Smooke GPT007 Rybens alielfilali01