MatTools: Benchmarking Large Language Models for Materials Science Tools
Large language models (LLMs) are increasingly applied to materials science
questions, including literature comprehension, property prediction, materials
discovery and alloy design. At the same time, a wide range of physics-based
computational approaches have been developed in which materials properties can
be calculated. Here, we propose a benchmark application to evaluate the
proficiency of LLMs to answer materials science questions through the
generation and safe execution of codes based on such physics-based
computational materials science packages. MatTools is built on two
complementary components: a materials simulation tool question-answer (QA)
benchmark and a real-world tool-usage benchmark. We designed an automated
methodology to efficiently collect real-world materials science tool-use
examples. The QA benchmark, derived from the pymatgen (Python Materials
Genomics) codebase and documentation, comprises 69,225 QA pairs that assess the
ability of an LLM to understand materials science tools. The real-world
benchmark contains 49 tasks (138 subtasks) requiring the generation of
functional Python code for materials property calculations. Our evaluation of
diverse LLMs yields three key insights: (1)Generalists outshine
specialists;(2)AI knows AI; and (3)Simpler is better. MatTools provides a
standardized framework for assessing and improving LLM capabilities for
materials science tool applications, facilitating the development of more
effective AI systems for materials science and general scientific research.
682a8dfbc8a4b019406bd8b0
Siyu Liu
claimed_verified
2025-05-19T06:42:45.155Z
6508469df3060ea840fd888f
https://cdn-avatars.huggingface.co/v1/production/uploads/6508469df3060ea840fd888f/3vwlKo0Cd9UWOPlwOvlJw.jpeg
Siyu Liu
user
SiyuLiu
682a8dfbc8a4b019406bd8b1
Jiamin Xu
admin_assigned
2025-05-19T14:22:06.474Z
656cf0757c934a7b3c2302c0
/avatars/4d4f585df63f0e2a23a56f9cbd067ba0.svg
Jiamin Xu
user
jiaminxu1019
682a8dfbc8a4b019406bd8b2
Beilin Ye
682a8dfbc8a4b019406bd8b3
Bo Hu
682a8dfbc8a4b019406bd8b4
David J. Srolovitz
682a8dfbc8a4b019406bd8b5
Tongqi Wen
MatTools: Benchmarking Large Language Models for Materials Science Tools
Large language models (LLMs) are increasingly applied to materials science
questions, including literature comprehension, property prediction, materials
discovery and alloy design. At the same time, a wide range of physics-based
computational approaches have been developed in which materials properties can
be calculated. Here, we propose a benchmark application to evaluate the
proficiency of LLMs to answer materials science questions through the
generation and safe execution of codes based on such physics-based
computational materials science packages. MatTools is built on two
complementary components: a materials simulation tool question-answer (QA)
benchmark and a real-world tool-usage benchmark. We designed an automated
methodology to efficiently collect real-world materials science tool-use
examples. The QA benchmark, derived from the pymatgen (Python Materials
Genomics) codebase and documentation, comprises 69,225 QA pairs that assess the
ability of an LLM to understand materials science tools. The real-world
benchmark contains 49 tasks (138 subtasks) requiring the generation of
functional Python code for materials property calculations. Our evaluation of
diverse LLMs yields three key insights: (1)Generalists outshine
specialists;(2)AI knows AI; and (3)Simpler is better. MatTools provides a
standardized framework for assessing and improving LLM capabilities for
materials science tool applications, facilitating the development of more
effective AI systems for materials science and general scientific research.
MatTools evaluates large language models' proficiency in materials science by assessing code generation and execution based on physics-based computational tools.
large language models LLMs materials science literature comprehension property prediction materials discovery alloy design physics-based computational models MatTools materials simulation tool question-answer QA benchmark pymatgen real-world tool-usage benchmark functional Python code materials property calculations