What indeed can GPT models do in chemistry? A comprehensive benchmark on
  eight tasks
Large Language Models (LLMs) with strong abilities in natural language
processing tasks have emerged and have been rapidly applied in various kinds of
areas such as science, finance and software engineering. However, the
capability of LLMs to advance the field of chemistry remains unclear. In this
paper,we establish a comprehensive benchmark containing 8 practical chemistry
tasks, including 1) name prediction, 2) property prediction, 3) yield
prediction, 4) reaction prediction, 5) retrosynthesis (prediction of reactants
from products), 6)text-based molecule design, 7) molecule captioning, and 8)
reagent selection. Our analysis draws on widely recognized datasets including
BBBP, Tox21, PubChem, USPTO, and ChEBI, facilitating a broad exploration of the
capacities of LLMs within the context of practical chemistry. Three GPT models
(GPT-4, GPT-3.5,and Davinci-003) are evaluated for each chemistry task in
zero-shot and few-shot in-context learning settings with carefully selected
demonstration examples and specially crafted prompts. The key results of our
investigation are 1) GPT-4 outperforms the other two models among the three
evaluated; 2) GPT models exhibit less competitive performance in tasks
demanding precise understanding of molecular SMILES representation, such as
reaction prediction and retrosynthesis;3) GPT models demonstrate strong
capabilities in text-related explanation tasks such as molecule captioning; and
4) GPT models exhibit comparable or better performance to classical machine
learning models when applied to chemical problems that can be transformed into
classification or ranking tasks, such as property prediction, and yield
prediction.
6476975c3d8d1cb791907c65
Taicheng Guo
claimed_verified
2023-06-05T10:07:38.050Z
60ed74f536ceac2554083559
/avatars/28c184ef76f719a720d933d05afb5800.svg
taicheng guo
user
taicheng
6476975c3d8d1cb791907c66
Kehan Guo
claimed_verified
2023-05-31T12:00:59.057Z
64770d4f5ef58684691e4e57
/avatars/7afe56ead9b1233b027e15d0e974f6af.svg
KehanGuo
user
kguo2
6476975c3d8d1cb791907c67
Bozhao nan
6476975c3d8d1cb791907c68
Zhengwen Liang
claimed_verified
2023-06-02T07:16:29.659Z
62ffa3f8311cad266f9af236
/avatars/203dac40bc546ee25a01d8715a4b3049.svg
Zhenwen Liang
user
invokerliang
6476975c3d8d1cb791907c69
Zhichun Guo
6476975c3d8d1cb791907c6a
Nitesh V. Chawla
6476975c3d8d1cb791907c6b
Olaf Wiest
6476975c3d8d1cb791907c6c
Xiangliang Zhang
What indeed can GPT models do in chemistry? A comprehensive benchmark on
  eight tasks
Large Language Models (LLMs) with strong abilities in natural language
processing tasks have emerged and have been rapidly applied in various kinds of
areas such as science, finance and software engineering. However, the
capability of LLMs to advance the field of chemistry remains unclear. In this
paper,we establish a comprehensive benchmark containing 8 practical chemistry
tasks, including 1) name prediction, 2) property prediction, 3) yield
prediction, 4) reaction prediction, 5) retrosynthesis (prediction of reactants
from products), 6)text-based molecule design, 7) molecule captioning, and 8)
reagent selection. Our analysis draws on widely recognized datasets including
BBBP, Tox21, PubChem, USPTO, and ChEBI, facilitating a broad exploration of the
capacities of LLMs within the context of practical chemistry. Three GPT models
(GPT-4, GPT-3.5,and Davinci-003) are evaluated for each chemistry task in
zero-shot and few-shot in-context learning settings with carefully selected
demonstration examples and specially crafted prompts. The key results of our
investigation are 1) GPT-4 outperforms the other two models among the three
evaluated; 2) GPT models exhibit less competitive performance in tasks
demanding precise understanding of molecular SMILES representation, such as
reaction prediction and retrosynthesis;3) GPT models demonstrate strong
capabilities in text-related explanation tasks such as molecule captioning; and
4) GPT models exhibit comparable or better performance to classical machine
learning models when applied to chemical problems that can be transformed into
classification or ranking tasks, such as property prediction, and yield
prediction.
Evaluation of GPT models on a comprehensive benchmark of practical chemistry tasks reveals varying performance across different types of chemical challenges.
GPT-4 GPT-3.5 Davinci-003 zero-shot few-shot in-context learning SMILES representation reaction prediction retrosynthesis molecule captioning property prediction yield prediction