content.type: new_line
content.raw: With the arrival of Twinkle April — Twinkle AI’s annual open-source celebration held every April — our community is excited to unveil its very first project:  
content.type: text
content.value: With the arrival of Twinkle April — Twinkle AI’s annual open-source celebration held every April — our community is excited to unveil its very first project:  
content.type: new_line
content.type: new_line
content.raw: 📊 Twinkle Eval (
content.type: text
content.value: 📊 Twinkle Eval (
content.href: https://github.com/ai-twinkle/Eval
content.raw: https://github.com/ai-twinkle/Eval
content.type: link
content.raw: ), a next-generation evaluation tool led by our contributor 
content.type: text
content.value: ), a next-generation evaluation tool led by our contributor 
content.raw: @tedslin
content.type: mention
content.user: tedslin
content.raw:  .
content.type: text
content.value:  .
content.type: new_line
content.type: new_line
content.raw: Unlike traditional evaluation tools like iKala’s ievals (
content.type: text
content.value: Unlike traditional evaluation tools like iKala’s ievals (
content.href: https://github.com/ikala-ai/ievals
content.raw: https://github.com/ikala-ai/ievals
content.type: link
content.raw: ), which can only evaluate language models (LMs) one sample at a time, Twinkle Eval is designed with Large Reasoning Models (LRMs) in mind. As reasoning time increases with more complex models, traditional tools become increasingly inefficient 😲 — for example, evaluating LRMs on the 
content.type: text
content.value: ), which can only evaluate language models (LMs) one sample at a time, Twinkle Eval is designed with Large Reasoning Models (LRMs) in mind. As reasoning time increases with more complex models, traditional tools become increasingly inefficient 😲 — for example, evaluating LRMs on the 
content.raw: https://huggingface.co/datasets/ikala/tmmluplus
content.id: ikala/tmmluplus
content.type: dataset
content.type: resource
content.url: https://huggingface.co/datasets/ikala/tmmluplus
content.raw:  benchmark could take *
content.type: text
content.value:  benchmark could take *
content.type: new_line
content.raw: half a day without finishing.
content.type: text
content.value: half a day without finishing.
content.type: new_line
content.type: new_line
content.raw: One question we were especially curious about:  
content.type: text
content.value: One question we were especially curious about:  
content.type: new_line
content.raw: Does shuffling multiple-choice answer order impact model accuracy? 🤔  
content.type: text
content.value: Does shuffling multiple-choice answer order impact model accuracy? 🤔  
content.type: new_line
content.raw: → See: "Change Answer Order Can Decrease MMLU Accuracy" – arXiv:2406.19470v1
content.type: text
content.value: → See: "Change Answer Order Can Decrease MMLU Accuracy" – arXiv:2406.19470v1
content.type: new_line
content.type: new_line
content.raw: To address these challenges, Twinkle Eval brings three key innovations to the table:
content.type: text
content.value: To address these challenges, Twinkle Eval brings three key innovations to the table:
content.type: new_line
content.type: new_line
content.raw: 1️⃣ Parallelized evaluation of samples  
content.type: text
content.value: 1️⃣ Parallelized evaluation of samples  
content.type: new_line
content.raw: 2️⃣ Multi-round testing for stability  
content.type: text
content.value: 2️⃣ Multi-round testing for stability  
content.type: new_line
content.raw: 3️⃣ Randomized answer order to test robustness  
content.type: text
content.value: 3️⃣ Randomized answer order to test robustness  
content.type: new_line
content.type: new_line
content.raw: After running experiments, we observed that Twinkle Eval can speed up evaluation by up to 15× 🚀🚀. Interestingly, most models scored slightly lower under the 2️⃣3️⃣ test settings compared to their claimed performance — suggesting further benchmarking is needed.
content.type: text
content.value: After running experiments, we observed that Twinkle Eval can speed up evaluation by up to 15× 🚀🚀. Interestingly, most models scored slightly lower under the 2️⃣3️⃣ test settings compared to their claimed performance — suggesting further benchmarking is needed.
content.type: new_line
content.type: new_line
content.raw: This framework also comes with additional tunable parameters and detailed logging of LM behavior per question — perfect for those who want to dive deeper. 😆
content.type: text
content.value: This framework also comes with additional tunable parameters and detailed logging of LM behavior per question — perfect for those who want to dive deeper. 😆
content.type: new_line
content.type: new_line
content.raw: If you find Twinkle Eval useful, please ⭐ the project and help spread the word 🤗
content.type: text
content.value: If you find Twinkle Eval useful, please ⭐ the project and help spread the word 🤗
rawContent: 
  With the arrival of Twinkle April — Twinkle AI’s annual open-source celebration held every April — our community is excited to unveil its very first project:  
  
  📊 Twinkle Eval (https://github.com/ai-twinkle/Eval), a next-generation evaluation tool led by our contributor @tedslin .
  
  Unlike traditional evaluation tools like iKala’s ievals (https://github.com/ikala-ai/ievals), which can only evaluate language models (LMs) one sample at a time, Twinkle Eval is designed with Large Reasoning Models (LRMs) in mind. As reasoning time increases with more complex models, traditional tools become increasingly inefficient 😲 — for example, evaluating LRMs on the https://huggingface.co/datasets/ikala/tmmluplus benchmark could take *
  half a day without finishing.
  
  One question we were especially curious about:  
  Does shuffling multiple-choice answer order impact model accuracy? 🤔  
  → See: "Change Answer Order Can Decrease MMLU Accuracy" – arXiv:2406.19470v1
  
  To address these challenges, Twinkle Eval brings three key innovations to the table:
  
  1️⃣ Parallelized evaluation of samples  
  2️⃣ Multi-round testing for stability  
  3️⃣ Randomized answer order to test robustness  
  
  After running experiments, we observed that Twinkle Eval can speed up evaluation by up to 15× 🚀🚀. Interestingly, most models scored slightly lower under the 2️⃣3️⃣ test settings compared to their claimed performance — suggesting further benchmarking is needed.
  
  This framework also comes with additional tunable parameters and detailed logging of LM behavior per question — perfect for those who want to dive deeper. 😆
  
  If you find Twinkle Eval useful, please ⭐ the project and help spread the word 🤗
reactions.reaction: 🔥
reactions.users: arpenxd John6666 dantezxcd
reactions.reaction: 👍
reactions.users: Vanos007
commentators._id: 67efb5e238e0880ff026e289
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/dPCQZ2Y4tNwXI0VmiDYI3.png
commentators.fullname: Febriyanto
commentators.name: arpenxd
commentators.type: user
commentators._id: 668a873c0bf195d6e5c1145a
commentators.avatarUrl: /avatars/81ed58c4996b33710ec8557eb676e546.svg
commentators.fullname: SajanGhimire
commentators.name: SajanGhi
commentators.type: user
commentators._id: 64bef7bc1363b5c799de6d44
commentators.avatarUrl: /avatars/a9947c6d7ca98d7385efa5ee7f2fb9a8.svg
commentators.fullname: hassenhamdi
commentators.name: hassenhamdi
commentators.type: user
commentators._id: 618dc56cbc345ca7bf95f3cd
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/618dc56cbc345ca7bf95f3cd/VTRlK8qY3zTV-b-LALHs4.png
commentators.fullname: Huang Liang Hsun
commentators.name: lianghsun
commentators.type: user