title: CURIE: Evaluating LLMs On Multitask Scientific Long Context
    Understanding and Reasoning
summary: Scientific problem-solving involves synthesizing information while applying
  expert knowledge. We introduce CURIE, a scientific long-Context
  Understanding,Reasoning and Information Extraction benchmark to measure the
  potential of Large Language Models (LLMs) in scientific problem-solving and
  assisting scientists in realistic workflows. This benchmark introduces ten
  challenging tasks with a total of 580 problems and solution pairs curated by
  experts in six disciplines - materials science, condensed matter physics,
  quantum computing, geospatial analysis, biodiversity, and proteins - covering
  both experimental and theoretical work-flows in science. We evaluate a range of
  closed and open LLMs on tasks in CURIE which requires domain expertise,
  comprehension of long in-context information,and multi-step reasoning. While
  Gemini Flash 2.0 and Claude-3 show consistent high comprehension across
  domains, the popular GPT-4o and command-R+ fail dramatically on protein
  sequencing tasks. With the best performance at 32% there is much room for
  improvement for all models. We hope that insights gained from CURIE can guide
  the future development of LLMs in sciences. Evaluation code and data are in
  https://github.com/google/curie
paper_authors._id: 67dc13a0b0451eaa2c5e298a
paper_authors.name: Hao Cui
paper_authors._id: 67dc13a0b0451eaa2c5e298b
paper_authors.name: Zahra Shamsi
paper_authors._id: 67dc13a0b0451eaa2c5e298c
paper_authors.name: Gowoon Cheon
paper_authors._id: 67dc13a0b0451eaa2c5e298d
paper_authors.name: Xuejian Ma
paper_authors._id: 67dc13a0b0451eaa2c5e298e
paper_authors.name: Shutong Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:31:25.035Z
paper_authors._id: 668ec66fd76c5aee9ba82a57
paper_authors.avatarUrl: /avatars/4ba953faaf05c483615d2ea1ae0b5572.svg
paper_authors.fullname: Shutong Li
paper_authors.type: user
paper_authors.user: LigeiaE
paper_authors._id: 67dc13a0b0451eaa2c5e298f
paper_authors.name: Maria Tikhanovskaya
paper_authors._id: 67dc13a0b0451eaa2c5e2990
paper_authors.name: Peter Norgaard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:31:16.168Z
paper_authors._id: 6751fd0c907b6ca2f6c71179
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/75-MmaXqwW_HFMDoHntkU.jpeg
paper_authors.fullname: Peter Norgaard
paper_authors.type: user
paper_authors.user: PNorgMagician
paper_authors._id: 67dc13a0b0451eaa2c5e2991
paper_authors.name: Nayantara Mudur
paper_authors._id: 67dc13a0b0451eaa2c5e2992
paper_authors.name: Martyna Plomecka
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:31:47.096Z
paper_authors._id: 6251ade7183aa426692442a0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6251ade7183aa426692442a0/1nP_mQqgcrSPgV1Cf2nMP.jpeg
paper_authors.fullname: Martyna Plomecka
paper_authors.type: user
paper_authors.user: Martynaplomecka
paper_authors._id: 67dc13a0b0451eaa2c5e2993
paper_authors.name: Paul Raccuglia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:31:53.506Z
paper_authors._id: 667c456320d68cc5814ebc04
paper_authors.avatarUrl: /avatars/4cb3e893f36aac3442e46afae9d725c7.svg
paper_authors.fullname: Paul Raccuglia
paper_authors.type: user
paper_authors.user: praccu-google
paper_authors._id: 67dc13a0b0451eaa2c5e2994
paper_authors.name: Yasaman Bahri
paper_authors._id: 67dc13a0b0451eaa2c5e2995
paper_authors.name: Victor V. Albert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:32:04.907Z
paper_authors._id: 66a7bb1d5f395fc0c6913f77
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sDFjAHabkD_SEYACVD5YU.jpeg
paper_authors.fullname: Victor V. Albert
paper_authors.type: user
paper_authors.user: valbert4
paper_authors._id: 67dc13a0b0451eaa2c5e2996
paper_authors.name: Pranesh Srinivasan
paper_authors._id: 67dc13a0b0451eaa2c5e2997
paper_authors.name: Haining Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:31:02.332Z
paper_authors._id: 64664b4ecf550af36eb40363
paper_authors.avatarUrl: /avatars/bf2b1ebbeac37f943578d5c93c834746.svg
paper_authors.fullname: Haining Pan
paper_authors.type: user
paper_authors.user: hainingpan
paper_authors._id: 67dc13a0b0451eaa2c5e2998
paper_authors.name: Philippe Faist
paper_authors._id: 67dc13a0b0451eaa2c5e2999
paper_authors.name: Brian Rohr
paper_authors._id: 67dc13a0b0451eaa2c5e299a
paper_authors.name: Michael J. Statt
paper_authors._id: 67dc13a0b0451eaa2c5e299b
paper_authors.name: Dan Morris
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:30:48.407Z
paper_authors._id: 6386a8ed06858a85f595b533
paper_authors.avatarUrl: /avatars/bdcd45db1dbc86d2f817bc682578cde3.svg
paper_authors.fullname: Dan Morris
paper_authors.type: user
paper_authors.user: danmorris427
paper_authors._id: 67dc13a0b0451eaa2c5e299c
paper_authors.name: Drew Purves
paper_authors._id: 67dc13a0b0451eaa2c5e299d
paper_authors.name: Elise Kleeman
paper_authors._id: 67dc13a0b0451eaa2c5e299e
paper_authors.name: Ruth Alcantara
paper_authors._id: 67dc13a0b0451eaa2c5e299f
paper_authors.name: Matthew Abraham
paper_authors._id: 67dc13a0b0451eaa2c5e29a0
paper_authors.name: Muqthar Mohammad
paper_authors._id: 67dc13a0b0451eaa2c5e29a1
paper_authors.name: Ean Phing VanLee
paper_authors._id: 67dc13a0b0451eaa2c5e29a2
paper_authors.name: Chenfei Jiang
paper_authors._id: 67dc13a0b0451eaa2c5e29a3
paper_authors.name: Elizabeth Dorfman
paper_authors._id: 67dc13a0b0451eaa2c5e29a4
paper_authors.name: Eun-Ah Kim
paper_authors._id: 67dc13a0b0451eaa2c5e29a5
paper_authors.name: Michael P Brenner
paper_authors._id: 67dc13a0b0451eaa2c5e29a6
paper_authors.name: Viren Jain
paper_authors._id: 67dc13a0b0451eaa2c5e29a7
paper_authors.name: Sameera Ponda
paper_authors._id: 67dc13a0b0451eaa2c5e29a8
paper_authors.name: Subhashini Venugopalan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-20T13:15:39.949Z
paper_authors._id: 646125a5933afb0106a9dabb
paper_authors.avatarUrl: /avatars/ccef359e4442f560a87fd66228e54c8b.svg
paper_authors.fullname: Subhashini
paper_authors.type: user
paper_authors.user: vsubhashini
paper_title: CURIE: Evaluating LLMs On Multitask Scientific Long Context
    Understanding and Reasoning
paper_summary: Scientific problem-solving involves synthesizing information while applying
  expert knowledge. We introduce CURIE, a scientific long-Context
  Understanding,Reasoning and Information Extraction benchmark to measure the
  potential of Large Language Models (LLMs) in scientific problem-solving and
  assisting scientists in realistic workflows. This benchmark introduces ten
  challenging tasks with a total of 580 problems and solution pairs curated by
  experts in six disciplines - materials science, condensed matter physics,
  quantum computing, geospatial analysis, biodiversity, and proteins - covering
  both experimental and theoretical work-flows in science. We evaluate a range of
  closed and open LLMs on tasks in CURIE which requires domain expertise,
  comprehension of long in-context information,and multi-step reasoning. While
  Gemini Flash 2.0 and Claude-3 show consistent high comprehension across
  domains, the popular GPT-4o and command-R+ fail dramatically on protein
  sequencing tasks. With the best performance at 32% there is much room for
  improvement for all models. We hope that insights gained from CURIE can guide
  the future development of LLMs in sciences. Evaluation code and data are in
  https://github.com/google/curie
paper_ai_summary: CURIE is a benchmark for evaluating Large Language Models (LLMs) in scientific problem-solving across various domains, highlighting areas of improvement for these models.
paper_ai_keywords: Large Language Models (LLMs) scientific problem-solving CURIE benchmark domain expertise comprehension multi-step reasoning protein sequencing tasks