title: SciCode: A Research Coding Benchmark Curated by Scientists
summary: Since language models (LMs) now outperform average humans on many challenging
  tasks, it has become increasingly difficult to develop challenging,
  high-quality, and realistic evaluations. We address this issue by examining
  LMs' capabilities to generate code for solving real scientific research
  problems. Incorporating input from scientists and AI researchers in 16 diverse
  natural science sub-fields, including mathematics, physics, chemistry, biology,
  and materials science, we created a scientist-curated coding benchmark,
  SciCode. The problems in SciCode naturally factorize into multiple subproblems,
  each involving knowledge recall, reasoning, and code synthesis. In total,
  SciCode contains 338 subproblems decomposed from 80 challenging main problems.
  It offers optional descriptions specifying useful scientific background
  information and scientist-annotated gold-standard solutions and test cases for
  evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can
  solve only 4.6% of the problems in the most realistic setting. We believe that
  SciCode demonstrates both contemporary LMs' progress towards becoming helpful
  scientific assistants and sheds light on the development and evaluation of
  scientific AI in the future.
paper_authors._id: 669dcf3b7f28c43e09d5497d
paper_authors.name: Minyang Tian
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-07-22T03:17:16.720Z
paper_authors._id: 64ead5f1213a0415bd22d0e4
paper_authors.avatarUrl: /avatars/2797e96d93b999a3e5f816935eb43673.svg
paper_authors.fullname: Minyang Tian
paper_authors.type: user
paper_authors.user: mtian8
paper_authors._id: 669dcf3b7f28c43e09d5497e
paper_authors.name: Luyu Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:12:44.059Z
paper_authors._id: 6009aa9126e23ab5edb47afd
paper_authors.avatarUrl: /avatars/f8afa9cf648227ad5817323451ede378.svg
paper_authors.fullname: Luyu Gao
paper_authors.type: user
paper_authors.user: Luyu
paper_authors._id: 669dcf3b7f28c43e09d5497f
paper_authors.name: Shizhuo Dylan Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-07-22T03:17:16.720Z
paper_authors._id: 660ec5a2509153ca49775a7c
paper_authors.avatarUrl: /avatars/97570fc245cc8ec7628da9c13bd35b71.svg
paper_authors.fullname: Hao Peng
paper_authors.type: user
paper_authors.user: haopeng01
paper_authors._id: 669dcf3b7f28c43e09d54980
paper_authors.name: Xinan Chen
paper_authors._id: 669dcf3b7f28c43e09d54981
paper_authors.name: Cunwei Fan
paper_authors._id: 669dcf3b7f28c43e09d54982
paper_authors.name: Xuefei Guo
paper_authors._id: 669dcf3b7f28c43e09d54983
paper_authors.name: Roland Haas
paper_authors._id: 669dcf3b7f28c43e09d54984
paper_authors.name: Pan Ji
paper_authors._id: 669dcf3b7f28c43e09d54985
paper_authors.name: Kittithat Krongchon
paper_authors._id: 669dcf3b7f28c43e09d54986
paper_authors.name: Yao Li
paper_authors._id: 669dcf3b7f28c43e09d54987
paper_authors.name: Shengyan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:31:26.403Z
paper_authors._id: 661551375e8f1eaaba4c53b3
paper_authors.avatarUrl: /avatars/1d41150d06ce4e4726269b9c48cbbc10.svg
paper_authors.fullname: LiuShengyang
paper_authors.type: user
paper_authors.user: LSY3579
paper_authors._id: 669dcf3b7f28c43e09d54988
paper_authors.name: Di Luo
paper_authors._id: 669dcf3b7f28c43e09d54989
paper_authors.name: Yutao Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:30:46.216Z
paper_authors._id: 630394e6eedc089484c367e5
paper_authors.avatarUrl: /avatars/26560adc1466cad5b5646d1a2aea3c76.svg
paper_authors.fullname: MA YUTAO
paper_authors.type: user
paper_authors.user: mytkkb
paper_authors._id: 669dcf3b7f28c43e09d5498a
paper_authors.name: Hao Tong
paper_authors._id: 669dcf3b7f28c43e09d5498b
paper_authors.name: Kha Trinh
paper_authors._id: 669dcf3b7f28c43e09d5498c
paper_authors.name: Chenyu Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:13:53.923Z
paper_authors._id: 624fa0a20f724e866aa833c6
paper_authors.avatarUrl: /avatars/be97e25a08b9a69200d8c7cc9a756410.svg
paper_authors.fullname: Chenyu Tian
paper_authors.type: user
paper_authors.user: CYBruce
paper_authors._id: 669dcf3b7f28c43e09d5498d
paper_authors.name: Zihan Wang
paper_authors._id: 669dcf3b7f28c43e09d5498e
paper_authors.name: Bohao Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T16:30:46.015Z
paper_authors._id: 6717b1a510c78be6e4335dd4
paper_authors.avatarUrl: /avatars/368e4af45151d2756726f45ec9cb0b64.svg
paper_authors.fullname: Bohao Wu
paper_authors.type: user
paper_authors.user: nlogn-27
paper_authors._id: 669dcf3b7f28c43e09d5498f
paper_authors.name: Yanyu Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:28:56.221Z
paper_authors._id: 645862428eef89b5c3c25ba1
paper_authors.avatarUrl: /avatars/b46887a16d17c6e552d26a5a140e81e0.svg
paper_authors.fullname: Yanyu Xiong
paper_authors.type: user
paper_authors.user: yxiong5
paper_authors._id: 669dcf3b7f28c43e09d54990
paper_authors.name: Shengzhu Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:28:47.804Z
paper_authors._id: 65306f8b2168c2bddd37930d
paper_authors.avatarUrl: /avatars/3c476272977cc7d6241dc1d6d63bd377.svg
paper_authors.fullname: Shengzhu Yin
paper_authors.type: user
paper_authors.user: shengzhu
paper_authors._id: 669dcf3b7f28c43e09d54991
paper_authors.name: Minhui Zhu
paper_authors._id: 669dcf3b7f28c43e09d54992
paper_authors.name: Kilian Lieret
paper_authors._id: 669dcf3b7f28c43e09d54993
paper_authors.name: Yanxin Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:08:39.863Z
paper_authors._id: 6610c90c6c59fff43622eef7
paper_authors.avatarUrl: /avatars/f832a3918101aa5f0834f1456b154c37.svg
paper_authors.fullname: Yanxin Lu
paper_authors.type: user
paper_authors.user: amber1120
paper_authors._id: 669dcf3b7f28c43e09d54994
paper_authors.name: Genglin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:16:25.059Z
paper_authors._id: 64881deb8e004bb92b0f4845
paper_authors.avatarUrl: /avatars/30a1e016d469bf7eb42c713351a9f65c.svg
paper_authors.fullname: Genglin Liu
paper_authors.type: user
paper_authors.user: genglinliu
paper_authors._id: 669dcf3b7f28c43e09d54995
paper_authors.name: Yufeng Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:16:33.054Z
paper_authors._id: 647e94f7770c299e56fc996f
paper_authors.avatarUrl: /avatars/6b53e9ece40e1ebd4a5297b47ebd8b91.svg
paper_authors.fullname: Yufeng Du
paper_authors.type: user
paper_authors.user: yufeng16
paper_authors._id: 669dcf3b7f28c43e09d54996
paper_authors.name: Tianhua Tao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:16:40.367Z
paper_authors._id: 61ecb698382b3b92fea7305f
paper_authors.avatarUrl: /avatars/b1bcf0d9ade12172810cb65dc9c4e4d8.svg
paper_authors.fullname: Tianhua Tao
paper_authors.type: user
paper_authors.user: Tianhua
paper_authors._id: 669dcf3b7f28c43e09d54997
paper_authors.name: Ofir Press
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:16:47.364Z
paper_authors._id: 65b7a5657817e067a5ad45d8
paper_authors.avatarUrl: /avatars/f8792352f16be80bd138ffc911138be1.svg
paper_authors.fullname: Ofir Press
paper_authors.type: user
paper_authors.user: ofirpress
paper_authors._id: 669dcf3b7f28c43e09d54998
paper_authors.name: Jamie Callan
paper_authors._id: 669dcf3b7f28c43e09d54999
paper_authors.name: Eliu Huerta
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-08T21:36:43.745Z
paper_authors._id: 677eef9d26d2ab07cfbcf6c6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/drlFgINAvi8M44TSME5-F.png
paper_authors.fullname: Eliu Huerta
paper_authors.type: user
paper_authors.user: elihu13
paper_authors._id: 669dcf3b7f28c43e09d5499a
paper_authors.name: Hao Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:18:14.706Z
paper_authors._id: 660ec5a2509153ca49775a7c
paper_authors.avatarUrl: /avatars/97570fc245cc8ec7628da9c13bd35b71.svg
paper_authors.fullname: Hao Peng
paper_authors.type: user
paper_authors.user: haopeng01
paper_title: SciCode: A Research Coding Benchmark Curated by Scientists
paper_summary: Since language models (LMs) now outperform average humans on many challenging
  tasks, it has become increasingly difficult to develop challenging,
  high-quality, and realistic evaluations. We address this issue by examining
  LMs' capabilities to generate code for solving real scientific research
  problems. Incorporating input from scientists and AI researchers in 16 diverse
  natural science sub-fields, including mathematics, physics, chemistry, biology,
  and materials science, we created a scientist-curated coding benchmark,
  SciCode. The problems in SciCode naturally factorize into multiple subproblems,
  each involving knowledge recall, reasoning, and code synthesis. In total,
  SciCode contains 338 subproblems decomposed from 80 challenging main problems.
  It offers optional descriptions specifying useful scientific background
  information and scientist-annotated gold-standard solutions and test cases for
  evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can
  solve only 4.6% of the problems in the most realistic setting. We believe that
  SciCode demonstrates both contemporary LMs' progress towards becoming helpful
  scientific assistants and sheds light on the development and evaluation of
  scientific AI in the future.
paper_ai_summary: SciCode, a scientist-curated coding benchmark, evaluates language models' ability to solve scientific research problems across various fields, highlighting current capabilities and future challenges in AI-assisted science.
paper_ai_keywords: language models coding benchmark knowledge recall reasoning code synthesis scientific AI