Changelog
Filter by MCP compatibility available in HF Spaces
•
54
Clémentine Fourrier
clefourrier
AI & ML interests
None yet
Recent Activity
new activity
about 5 hours ago
gaia-benchmark/GAIA:Question: 99c9cc74-fdc8-46c6-8f8d-3ce2d3bfeea3
upvoted
a
changelog
about 6 hours ago
Filter by MCP compatibility available in HF Spaces
upvoted
a
changelog
about 6 hours ago
AI-generated Abstract summaries on Hugging Face Papers
Organizations
clefourrier's activity
Question: 99c9cc74-fdc8-46c6-8f8d-3ce2d3bfeea3
1
#25 opened about 6 hours ago
by
artyomboyko
upvoted
3
changelogs
about 6 hours ago
Changelog
AI-generated Abstract summaries on Hugging Face Papers
•
43
Changelog
Static Spaces can now have a build step
•
21
posted
an
update
about 6 hours ago
Post
70
Saying Claude 4 is "the best coding model in the world" from the SWEBench scores is super misleading, and here is why:
If you look at the announcement table, their model has the best scores, but... if you look at the very bottom, in font 4, you'll see that the metric they report is actually not the same metric as the one used for the other models!
Comparing "pass@1 averaged 10 times" to "normal pass@1" is like grading one student by allowing them to take the test 10 times and averaging question scores, when the other students only get one chance at grading.
The first way to grade (avg@10) is actually quite good statistically, much better than what model creators usually report, because models tend to be quite inconsistent - sometimes good, sometimes bad...
But! You want to do it for all models then, and report with error bars.
The issue is that, if you do... well, it's going to be harder to say your model is the best, because the error bars will overlap between models, by a lot.
Also, you'll see that 2 numbers are reported: the first one is using avg@10 (what I explained above), and the second, highest one is using this plus many other tricks:
- test time compute (so having the model generate a tree of answers and selecting the best as you go, more or less)
- removing the times when the model breaks the tests
- and using another model to select the most promising solution!
You can't really say it's better than the rest, mostly because it's **way less efficient** to achieve a similar result.
It's honestly a bit sad because from user reports, the model sounds good - however, this announcement is overblown numbers wise, and I'm quite sure it's more a problem of "too much marketing" than of "bad science"
Another thing which makes the comparison invalid is the complete absence of open source from the report - don't think they are aware of DeepSeek/ Qwen/The new mistral for code/and all the cool specialised models found on the hub?
If you look at the announcement table, their model has the best scores, but... if you look at the very bottom, in font 4, you'll see that the metric they report is actually not the same metric as the one used for the other models!
Comparing "pass@1 averaged 10 times" to "normal pass@1" is like grading one student by allowing them to take the test 10 times and averaging question scores, when the other students only get one chance at grading.
The first way to grade (avg@10) is actually quite good statistically, much better than what model creators usually report, because models tend to be quite inconsistent - sometimes good, sometimes bad...
But! You want to do it for all models then, and report with error bars.
The issue is that, if you do... well, it's going to be harder to say your model is the best, because the error bars will overlap between models, by a lot.
Also, you'll see that 2 numbers are reported: the first one is using avg@10 (what I explained above), and the second, highest one is using this plus many other tricks:
- test time compute (so having the model generate a tree of answers and selecting the best as you go, more or less)
- removing the times when the model breaks the tests
- and using another model to select the most promising solution!
You can't really say it's better than the rest, mostly because it's **way less efficient** to achieve a similar result.
It's honestly a bit sad because from user reports, the model sounds good - however, this announcement is overblown numbers wise, and I'm quite sure it's more a problem of "too much marketing" than of "bad science"
Another thing which makes the comparison invalid is the complete absence of open source from the report - don't think they are aware of DeepSeek/ Qwen/The new mistral for code/and all the cool specialised models found on the hub?
reacted to
fdaudens's
post with ❤️
about 6 hours ago
Post
194
Here’s what happens when a national institution builds its own digital intelligence: France’s Ministry of Culture just released 17K+ real users testing 30+ chatbots in French. Raw, diverse, and a goldmine for studying LLMs in the wild.
ministere-culture/comparia-conversations
ministere-culture/comparia-conversations
updated
a
dataset
about 7 hours ago
replied to
their
post
about 10 hours ago
Hi! The official GAIA leaderboard is unrelated to the test leaderboard for the agents classes :)
You should contact
@burtenshaw
who's managing the classes courses :)
updated
a
dataset
about 13 hours ago
updated
a
dataset
about 17 hours ago
updated
2
datasets
about 19 hours ago
updated
a
dataset
1 day ago