title: Are large language models superhuman chemists?
summary: Large language models (LLMs) have gained widespread interest due to their
  ability to process human language and perform tasks on which they have not been
  explicitly trained. This is relevant for the chemical sciences, which face the
  problem of small and diverse datasets that are frequently in the form of text.
  LLMs have shown promise in addressing these issues and are increasingly being
  harnessed to predict chemical properties, optimize reactions, and even design
  and conduct experiments autonomously. However, we still have only a very
  limited systematic understanding of the chemical reasoning capabilities of
  LLMs, which would be required to improve models and mitigate potential harms.
  Here, we introduce "ChemBench," an automated framework designed to rigorously
  evaluate the chemical knowledge and reasoning abilities of state-of-the-art
  LLMs against the expertise of human chemists. We curated more than 7,000
  question-answer pairs for a wide array of subfields of the chemical sciences,
  evaluated leading open and closed-source LLMs, and found that the best models
  outperformed the best human chemists in our study on average. The models,
  however, struggle with some chemical reasoning tasks that are easy for human
  experts and provide overconfident, misleading predictions, such as about
  chemicals' safety profiles. These findings underscore the dual reality that,
  although LLMs demonstrate remarkable proficiency in chemical tasks, further
  research is critical to enhancing their safety and utility in chemical
  sciences. Our findings also indicate a need for adaptations to chemistry
  curricula and highlight the importance of continuing to develop evaluation
  frameworks to improve safe and useful LLMs.
paper_authors._id: 660cbe6099760c2ef3b1a9eb
paper_authors.name: Adrian Mirza
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-13T07:49:29.699Z
paper_authors._id: 64dde64b8276275de53faf79
paper_authors.avatarUrl: /avatars/32b438efaca7b83b8923b5ec34880faf.svg
paper_authors.fullname: Adrian Mirza
paper_authors.type: user
paper_authors.user: AdrianM0
paper_authors._id: 660cbe6099760c2ef3b1a9ec
paper_authors.name: Nawaf Alampara
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-03T18:26:34.228Z
paper_authors._id: 651dbb2f56b71e7dbc9a95c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651dbb2f56b71e7dbc9a95c4/hD4OJnyEruOJF_2KaiF1D.png
paper_authors.fullname: Nawaf Alampara
paper_authors.type: user
paper_authors.user: n0w0f
paper_authors._id: 660cbe6099760c2ef3b1a9ed
paper_authors.name: Sreekanth Kunchapu
paper_authors._id: 660cbe6099760c2ef3b1a9ee
paper_authors.name: Benedict Emoekabu
paper_authors._id: 660cbe6099760c2ef3b1a9ef
paper_authors.name: Aswanth Krishnan
paper_authors._id: 660cbe6099760c2ef3b1a9f0
paper_authors.name: Mara Wilhelmi
paper_authors._id: 660cbe6099760c2ef3b1a9f1
paper_authors.name: Macjonathan Okereke
paper_authors._id: 660cbe6099760c2ef3b1a9f2
paper_authors.name: Juliane Eberhardt
paper_authors._id: 660cbe6099760c2ef3b1a9f3
paper_authors.name: Amir Mohammad Elahi
paper_authors._id: 660cbe6099760c2ef3b1a9f4
paper_authors.name: Maximilian Greiner
paper_authors._id: 660cbe6099760c2ef3b1a9f5
paper_authors.name: Caroline T. Holick
paper_authors._id: 660cbe6099760c2ef3b1a9f6
paper_authors.name: Tanya Gupta
paper_authors._id: 660cbe6099760c2ef3b1a9f7
paper_authors.name: Mehrdad Asgari
paper_authors._id: 660cbe6099760c2ef3b1a9f8
paper_authors.name: Christina Glaubitz
paper_authors._id: 660cbe6099760c2ef3b1a9f9
paper_authors.name: Lea C. Klepsch
paper_authors._id: 660cbe6099760c2ef3b1a9fa
paper_authors.name: Yannik Köster
paper_authors._id: 660cbe6099760c2ef3b1a9fb
paper_authors.name: Jakob Meyer
paper_authors._id: 660cbe6099760c2ef3b1a9fc
paper_authors.name: Santiago Miret
paper_authors._id: 660cbe6099760c2ef3b1a9fd
paper_authors.name: Tim Hoffmann
paper_authors._id: 660cbe6099760c2ef3b1a9fe
paper_authors.name: Fabian Alexander Kreth
paper_authors._id: 660cbe6099760c2ef3b1a9ff
paper_authors.name: Michael Ringleb
paper_authors._id: 660cbe6099760c2ef3b1aa00
paper_authors.name: Nicole Roesner
paper_authors._id: 660cbe6099760c2ef3b1aa01
paper_authors.name: Ulrich S. Schubert
paper_authors._id: 660cbe6099760c2ef3b1aa02
paper_authors.name: Leanne M. Stafast
paper_authors._id: 660cbe6099760c2ef3b1aa03
paper_authors.name: Dinga Wonanke
paper_authors._id: 660cbe6099760c2ef3b1aa04
paper_authors.name: Michael Pieler
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-24T11:12:17.385Z
paper_authors._id: 6037f54f770949ef34f12d99
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1614280013027-noauth.jpeg
paper_authors.fullname: Michael Pieler
paper_authors.type: user
paper_authors.user: MicPie
paper_authors._id: 660cbe6099760c2ef3b1aa05
paper_authors.name: Philippe Schwaller
paper_authors._id: 660cbe6099760c2ef3b1aa06
paper_authors.name: Kevin Maik Jablonka
paper_title: Are large language models superhuman chemists?
paper_summary: Large language models (LLMs) have gained widespread interest due to their
  ability to process human language and perform tasks on which they have not been
  explicitly trained. This is relevant for the chemical sciences, which face the
  problem of small and diverse datasets that are frequently in the form of text.
  LLMs have shown promise in addressing these issues and are increasingly being
  harnessed to predict chemical properties, optimize reactions, and even design
  and conduct experiments autonomously. However, we still have only a very
  limited systematic understanding of the chemical reasoning capabilities of
  LLMs, which would be required to improve models and mitigate potential harms.
  Here, we introduce "ChemBench," an automated framework designed to rigorously
  evaluate the chemical knowledge and reasoning abilities of state-of-the-art
  LLMs against the expertise of human chemists. We curated more than 7,000
  question-answer pairs for a wide array of subfields of the chemical sciences,
  evaluated leading open and closed-source LLMs, and found that the best models
  outperformed the best human chemists in our study on average. The models,
  however, struggle with some chemical reasoning tasks that are easy for human
  experts and provide overconfident, misleading predictions, such as about
  chemicals' safety profiles. These findings underscore the dual reality that,
  although LLMs demonstrate remarkable proficiency in chemical tasks, further
  research is critical to enhancing their safety and utility in chemical
  sciences. Our findings also indicate a need for adaptations to chemistry
  curricula and highlight the importance of continuing to develop evaluation
  frameworks to improve safe and useful LLMs.
paper_ai_summary: ChemBench, an automated evaluation framework, assesses the chemical reasoning of large language models against human experts, revealing their proficiency and shortcomings in specific tasks.
paper_ai_keywords: large language models LLMs chemical sciences datasets text chemical properties reactions experiments chemical reasoning ChemBench question-answer pairs subfields chemical knowledge human chemists safety profiles adaptations curricula evaluation frameworks