SciArena: An Open Evaluation Platform for Foundation Models in
  Scientific Literature Tasks
We present SciArena, an open and collaborative platform for evaluating
foundation models on scientific literature tasks. Unlike traditional benchmarks
for scientific literature understanding and synthesis, SciArena engages the
research community directly, following the Chatbot Arena evaluation approach of
community voting on model comparisons. By leveraging collective intelligence,
SciArena offers a community-driven evaluation of model performance on
open-ended scientific tasks that demand literature-grounded, long-form
responses. The platform currently supports 23 open-source and proprietary
foundation models and has collected over 13,000 votes from trusted researchers
across diverse scientific domains. We analyze the data collected so far and
confirm that the submitted questions are diverse, aligned with real-world
literature needs, and that participating researchers demonstrate strong
self-consistency and inter-annotator agreement in their evaluations. We discuss
the results and insights based on the model ranking leaderboard. To further
promote research in building model-based automated evaluation systems for
literature tasks, we release SciArena-Eval, a meta-evaluation benchmark based
on our collected preference data. The benchmark measures the accuracy of models
in judging answer quality by comparing their pairwise assessments with human
votes. Our experiments highlight the benchmark's challenges and emphasize the
need for more reliable automated evaluation methods.
6864a4ddd59a9eda59024aea
Yilun Zhao
claimed_verified
2025-07-02T07:38:54.448Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
6864a4ddd59a9eda59024aeb
Kaiyan Zhang
claimed_verified
2025-07-02T07:38:52.477Z
676ce7767fff9075b5d526fa
/avatars/bf6697163b91564a8d4b773d3f6420bf.svg
Kaiyan Zhang
user
maxzky
6864a4ddd59a9eda59024aec
Tiansheng Hu
claimed_verified
2025-07-02T07:38:50.408Z
67492b9e347c3876f22b3684
/avatars/0d80d23f7b10ce8bac689f6e8317a014.svg
Tiansheng Hu
user
HughieHu
6864a4ddd59a9eda59024aed
Sihong Wu
claimed_verified
2025-07-02T16:07:51.232Z
67911b57692e0b97d9936ee3
/avatars/270cb16773b0ba2a1fde8e7e18979971.svg
Sihong Wu
user
Primrose255
6864a4ddd59a9eda59024aee
Ronan Le Bras
6864a4ddd59a9eda59024aef
Taira Anderson
6864a4ddd59a9eda59024af0
Jonathan Bragg
6864a4ddd59a9eda59024af1
Joseph Chee Chang
6864a4ddd59a9eda59024af2
Jesse Dodge
6864a4ddd59a9eda59024af3
Matt Latzke
6864a4ddd59a9eda59024af4
Yixin Liu
6864a4ddd59a9eda59024af5
Charles McGrady
6864a4ddd59a9eda59024af6
Xiangru Tang
6864a4ddd59a9eda59024af7
Zihang Wang
claimed_verified
2025-07-02T12:23:36.042Z
65d6d12012e036f74a08ce75
/avatars/27ccbeb7c6f7d67ad7a6eed744e92a37.svg
Zihang Wang
user
zihang93
6864a4ddd59a9eda59024af8
Chen Zhao
6864a4ddd59a9eda59024af9
Hannaneh Hajishirzi
6864a4ddd59a9eda59024afa
Doug Downey
6864a4ddd59a9eda59024afb
Arman Cohan
SciArena: An Open Evaluation Platform for Foundation Models in
  Scientific Literature Tasks
We present SciArena, an open and collaborative platform for evaluating
foundation models on scientific literature tasks. Unlike traditional benchmarks
for scientific literature understanding and synthesis, SciArena engages the
research community directly, following the Chatbot Arena evaluation approach of
community voting on model comparisons. By leveraging collective intelligence,
SciArena offers a community-driven evaluation of model performance on
open-ended scientific tasks that demand literature-grounded, long-form
responses. The platform currently supports 23 open-source and proprietary
foundation models and has collected over 13,000 votes from trusted researchers
across diverse scientific domains. We analyze the data collected so far and
confirm that the submitted questions are diverse, aligned with real-world
literature needs, and that participating researchers demonstrate strong
self-consistency and inter-annotator agreement in their evaluations. We discuss
the results and insights based on the model ranking leaderboard. To further
promote research in building model-based automated evaluation systems for
literature tasks, we release SciArena-Eval, a meta-evaluation benchmark based
on our collected preference data. The benchmark measures the accuracy of models
in judging answer quality by comparing their pairwise assessments with human
votes. Our experiments highlight the benchmark's challenges and emphasize the
need for more reliable automated evaluation methods.
SciArena is a community-driven platform for evaluating foundation models on scientific literature tasks using collective intelligence and human voting.
Chatbot Arena foundation models scientific literature understanding scientific literature synthesis open-ended scientific tasks literature-grounded long-form responses community-driven evaluation model ranking leaderboard meta-evaluation benchmark automated evaluation systems