new_line
With the arrival of Twinkle April — Twinkle AI’s annual open-source celebration held every April — our community is excited to unveil its very first project:  
text
With the arrival of Twinkle April — Twinkle AI’s annual open-source celebration held every April — our community is excited to unveil its very first project:  
new_line
new_line
📊 Twinkle Eval (
text
📊 Twinkle Eval (
https://github.com/ai-twinkle/Eval
https://github.com/ai-twinkle/Eval
link
), a next-generation evaluation tool led by our contributor 
text
), a next-generation evaluation tool led by our contributor 
@tedslin
mention
tedslin
 .
text
 .
new_line
new_line
Unlike traditional evaluation tools like iKala’s ievals (
text
Unlike traditional evaluation tools like iKala’s ievals (
https://github.com/ikala-ai/ievals
https://github.com/ikala-ai/ievals
link
), which can only evaluate language models (LMs) one sample at a time, Twinkle Eval is designed with Large Reasoning Models (LRMs) in mind. As reasoning time increases with more complex models, traditional tools become increasingly inefficient 😲 — for example, evaluating LRMs on the 
text
), which can only evaluate language models (LMs) one sample at a time, Twinkle Eval is designed with Large Reasoning Models (LRMs) in mind. As reasoning time increases with more complex models, traditional tools become increasingly inefficient 😲 — for example, evaluating LRMs on the 
https://huggingface.co/datasets/ikala/tmmluplus
ikala/tmmluplus
dataset
resource
https://huggingface.co/datasets/ikala/tmmluplus
 benchmark could take *
text
 benchmark could take *
new_line
half a day without finishing.
text
half a day without finishing.
new_line
new_line
One question we were especially curious about:  
text
One question we were especially curious about:  
new_line
Does shuffling multiple-choice answer order impact model accuracy? 🤔  
text
Does shuffling multiple-choice answer order impact model accuracy? 🤔  
new_line
→ See: "Change Answer Order Can Decrease MMLU Accuracy" – arXiv:2406.19470v1
text
→ See: "Change Answer Order Can Decrease MMLU Accuracy" – arXiv:2406.19470v1
new_line
new_line
To address these challenges, Twinkle Eval brings three key innovations to the table:
text
To address these challenges, Twinkle Eval brings three key innovations to the table:
new_line
new_line
1️⃣ Parallelized evaluation of samples  
text
1️⃣ Parallelized evaluation of samples  
new_line
2️⃣ Multi-round testing for stability  
text
2️⃣ Multi-round testing for stability  
new_line
3️⃣ Randomized answer order to test robustness  
text
3️⃣ Randomized answer order to test robustness  
new_line
new_line
After running experiments, we observed that Twinkle Eval can speed up evaluation by up to 15× 🚀🚀. Interestingly, most models scored slightly lower under the 2️⃣3️⃣ test settings compared to their claimed performance — suggesting further benchmarking is needed.
text
After running experiments, we observed that Twinkle Eval can speed up evaluation by up to 15× 🚀🚀. Interestingly, most models scored slightly lower under the 2️⃣3️⃣ test settings compared to their claimed performance — suggesting further benchmarking is needed.
new_line
new_line
This framework also comes with additional tunable parameters and detailed logging of LM behavior per question — perfect for those who want to dive deeper. 😆
text
This framework also comes with additional tunable parameters and detailed logging of LM behavior per question — perfect for those who want to dive deeper. 😆
new_line
new_line
If you find Twinkle Eval useful, please ⭐ the project and help spread the word 🤗
text
If you find Twinkle Eval useful, please ⭐ the project and help spread the word 🤗

With the arrival of Twinkle April — Twinkle AI’s annual open-source celebration held every April — our community is excited to unveil its very first project:  

📊 Twinkle Eval (https://github.com/ai-twinkle/Eval), a next-generation evaluation tool led by our contributor @tedslin .

Unlike traditional evaluation tools like iKala’s ievals (https://github.com/ikala-ai/ievals), which can only evaluate language models (LMs) one sample at a time, Twinkle Eval is designed with Large Reasoning Models (LRMs) in mind. As reasoning time increases with more complex models, traditional tools become increasingly inefficient 😲 — for example, evaluating LRMs on the https://huggingface.co/datasets/ikala/tmmluplus benchmark could take *
half a day without finishing.

One question we were especially curious about:  
Does shuffling multiple-choice answer order impact model accuracy? 🤔  
→ See: "Change Answer Order Can Decrease MMLU Accuracy" – arXiv:2406.19470v1

To address these challenges, Twinkle Eval brings three key innovations to the table:

1️⃣ Parallelized evaluation of samples  
2️⃣ Multi-round testing for stability  
3️⃣ Randomized answer order to test robustness  

After running experiments, we observed that Twinkle Eval can speed up evaluation by up to 15× 🚀🚀. Interestingly, most models scored slightly lower under the 2️⃣3️⃣ test settings compared to their claimed performance — suggesting further benchmarking is needed.

This framework also comes with additional tunable parameters and detailed logging of LM behavior per question — perfect for those who want to dive deeper. 😆

If you find Twinkle Eval useful, please ⭐ the project and help spread the word 🤗
🔥
arpenxd John6666 dantezxcd
👍
Vanos007
67efb5e238e0880ff026e289
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/dPCQZ2Y4tNwXI0VmiDYI3.png
Febriyanto
arpenxd
user
668a873c0bf195d6e5c1145a
/avatars/81ed58c4996b33710ec8557eb676e546.svg
SajanGhimire
SajanGhi
user
64bef7bc1363b5c799de6d44
/avatars/a9947c6d7ca98d7385efa5ee7f2fb9a8.svg
hassenhamdi
hassenhamdi
user
618dc56cbc345ca7bf95f3cd
https://cdn-avatars.huggingface.co/v1/production/uploads/618dc56cbc345ca7bf95f3cd/VTRlK8qY3zTV-b-LALHs4.png
Huang Liang Hsun
lianghsun
user