title: What indeed can GPT models do in chemistry? A comprehensive benchmark on
    eight tasks
summary: Large Language Models (LLMs) with strong abilities in natural language
  processing tasks have emerged and have been rapidly applied in various kinds of
  areas such as science, finance and software engineering. However, the
  capability of LLMs to advance the field of chemistry remains unclear. In this
  paper,we establish a comprehensive benchmark containing 8 practical chemistry
  tasks, including 1) name prediction, 2) property prediction, 3) yield
  prediction, 4) reaction prediction, 5) retrosynthesis (prediction of reactants
  from products), 6)text-based molecule design, 7) molecule captioning, and 8)
  reagent selection. Our analysis draws on widely recognized datasets including
  BBBP, Tox21, PubChem, USPTO, and ChEBI, facilitating a broad exploration of the
  capacities of LLMs within the context of practical chemistry. Three GPT models
  (GPT-4, GPT-3.5,and Davinci-003) are evaluated for each chemistry task in
  zero-shot and few-shot in-context learning settings with carefully selected
  demonstration examples and specially crafted prompts. The key results of our
  investigation are 1) GPT-4 outperforms the other two models among the three
  evaluated; 2) GPT models exhibit less competitive performance in tasks
  demanding precise understanding of molecular SMILES representation, such as
  reaction prediction and retrosynthesis;3) GPT models demonstrate strong
  capabilities in text-related explanation tasks such as molecule captioning; and
  4) GPT models exhibit comparable or better performance to classical machine
  learning models when applied to chemical problems that can be transformed into
  classification or ranking tasks, such as property prediction, and yield
  prediction.
paper_authors._id: 6476975c3d8d1cb791907c65
paper_authors.name: Taicheng Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:07:38.050Z
paper_authors._id: 60ed74f536ceac2554083559
paper_authors.avatarUrl: /avatars/28c184ef76f719a720d933d05afb5800.svg
paper_authors.fullname: taicheng guo
paper_authors.type: user
paper_authors.user: taicheng
paper_authors._id: 6476975c3d8d1cb791907c66
paper_authors.name: Kehan Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-31T12:00:59.057Z
paper_authors._id: 64770d4f5ef58684691e4e57
paper_authors.avatarUrl: /avatars/7afe56ead9b1233b027e15d0e974f6af.svg
paper_authors.fullname: KehanGuo
paper_authors.type: user
paper_authors.user: kguo2
paper_authors._id: 6476975c3d8d1cb791907c67
paper_authors.name: Bozhao nan
paper_authors._id: 6476975c3d8d1cb791907c68
paper_authors.name: Zhengwen Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-02T07:16:29.659Z
paper_authors._id: 62ffa3f8311cad266f9af236
paper_authors.avatarUrl: /avatars/4c88cb518e000a475f8381573f21aa7f.svg
paper_authors.fullname: Zhenwen Liang
paper_authors.type: user
paper_authors.user: invokerliang
paper_authors._id: 6476975c3d8d1cb791907c69
paper_authors.name: Zhichun Guo
paper_authors._id: 6476975c3d8d1cb791907c6a
paper_authors.name: Nitesh V. Chawla
paper_authors._id: 6476975c3d8d1cb791907c6b
paper_authors.name: Olaf Wiest
paper_authors._id: 6476975c3d8d1cb791907c6c
paper_authors.name: Xiangliang Zhang
paper_title: What indeed can GPT models do in chemistry? A comprehensive benchmark on
    eight tasks
paper_summary: Large Language Models (LLMs) with strong abilities in natural language
  processing tasks have emerged and have been rapidly applied in various kinds of
  areas such as science, finance and software engineering. However, the
  capability of LLMs to advance the field of chemistry remains unclear. In this
  paper,we establish a comprehensive benchmark containing 8 practical chemistry
  tasks, including 1) name prediction, 2) property prediction, 3) yield
  prediction, 4) reaction prediction, 5) retrosynthesis (prediction of reactants
  from products), 6)text-based molecule design, 7) molecule captioning, and 8)
  reagent selection. Our analysis draws on widely recognized datasets including
  BBBP, Tox21, PubChem, USPTO, and ChEBI, facilitating a broad exploration of the
  capacities of LLMs within the context of practical chemistry. Three GPT models
  (GPT-4, GPT-3.5,and Davinci-003) are evaluated for each chemistry task in
  zero-shot and few-shot in-context learning settings with carefully selected
  demonstration examples and specially crafted prompts. The key results of our
  investigation are 1) GPT-4 outperforms the other two models among the three
  evaluated; 2) GPT models exhibit less competitive performance in tasks
  demanding precise understanding of molecular SMILES representation, such as
  reaction prediction and retrosynthesis;3) GPT models demonstrate strong
  capabilities in text-related explanation tasks such as molecule captioning; and
  4) GPT models exhibit comparable or better performance to classical machine
  learning models when applied to chemical problems that can be transformed into
  classification or ranking tasks, such as property prediction, and yield
  prediction.
paper_ai_summary: Evaluation of GPT models on a comprehensive benchmark of practical chemistry tasks reveals varying performance across different types of chemical challenges.
paper_ai_keywords: GPT-4 GPT-3.5 Davinci-003 zero-shot few-shot in-context learning SMILES representation reaction prediction retrosynthesis molecule captioning property prediction yield prediction