title: MatTools: Benchmarking Large Language Models for Materials Science Tools
summary: Large language models (LLMs) are increasingly applied to materials science
  questions, including literature comprehension, property prediction, materials
  discovery and alloy design. At the same time, a wide range of physics-based
  computational approaches have been developed in which materials properties can
  be calculated. Here, we propose a benchmark application to evaluate the
  proficiency of LLMs to answer materials science questions through the
  generation and safe execution of codes based on such physics-based
  computational materials science packages. MatTools is built on two
  complementary components: a materials simulation tool question-answer (QA)
  benchmark and a real-world tool-usage benchmark. We designed an automated
  methodology to efficiently collect real-world materials science tool-use
  examples. The QA benchmark, derived from the pymatgen (Python Materials
  Genomics) codebase and documentation, comprises 69,225 QA pairs that assess the
  ability of an LLM to understand materials science tools. The real-world
  benchmark contains 49 tasks (138 subtasks) requiring the generation of
  functional Python code for materials property calculations. Our evaluation of
  diverse LLMs yields three key insights: (1)Generalists outshine
  specialists;(2)AI knows AI; and (3)Simpler is better. MatTools provides a
  standardized framework for assessing and improving LLM capabilities for
  materials science tool applications, facilitating the development of more
  effective AI systems for materials science and general scientific research.
paper_authors._id: 682a8dfbc8a4b019406bd8b0
paper_authors.name: Siyu Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T06:42:45.155Z
paper_authors._id: 6508469df3060ea840fd888f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6508469df3060ea840fd888f/3vwlKo0Cd9UWOPlwOvlJw.jpeg
paper_authors.fullname: Siyu Liu
paper_authors.type: user
paper_authors.user: SiyuLiu
paper_authors._id: 682a8dfbc8a4b019406bd8b1
paper_authors.name: Jiamin Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-19T14:22:06.474Z
paper_authors._id: 656cf0757c934a7b3c2302c0
paper_authors.avatarUrl: /avatars/4d4f585df63f0e2a23a56f9cbd067ba0.svg
paper_authors.fullname: Jiamin Xu
paper_authors.type: user
paper_authors.user: jiaminxu1019
paper_authors._id: 682a8dfbc8a4b019406bd8b2
paper_authors.name: Beilin Ye
paper_authors._id: 682a8dfbc8a4b019406bd8b3
paper_authors.name: Bo Hu
paper_authors._id: 682a8dfbc8a4b019406bd8b4
paper_authors.name: David J. Srolovitz
paper_authors._id: 682a8dfbc8a4b019406bd8b5
paper_authors.name: Tongqi Wen
paper_title: MatTools: Benchmarking Large Language Models for Materials Science Tools
paper_summary: Large language models (LLMs) are increasingly applied to materials science
  questions, including literature comprehension, property prediction, materials
  discovery and alloy design. At the same time, a wide range of physics-based
  computational approaches have been developed in which materials properties can
  be calculated. Here, we propose a benchmark application to evaluate the
  proficiency of LLMs to answer materials science questions through the
  generation and safe execution of codes based on such physics-based
  computational materials science packages. MatTools is built on two
  complementary components: a materials simulation tool question-answer (QA)
  benchmark and a real-world tool-usage benchmark. We designed an automated
  methodology to efficiently collect real-world materials science tool-use
  examples. The QA benchmark, derived from the pymatgen (Python Materials
  Genomics) codebase and documentation, comprises 69,225 QA pairs that assess the
  ability of an LLM to understand materials science tools. The real-world
  benchmark contains 49 tasks (138 subtasks) requiring the generation of
  functional Python code for materials property calculations. Our evaluation of
  diverse LLMs yields three key insights: (1)Generalists outshine
  specialists;(2)AI knows AI; and (3)Simpler is better. MatTools provides a
  standardized framework for assessing and improving LLM capabilities for
  materials science tool applications, facilitating the development of more
  effective AI systems for materials science and general scientific research.
paper_ai_summary: MatTools evaluates large language models' proficiency in materials science by assessing code generation and execution based on physics-based computational tools.
paper_ai_keywords: large language models LLMs materials science literature comprehension property prediction materials discovery alloy design physics-based computational models MatTools materials simulation tool question-answer QA benchmark pymatgen real-world tool-usage benchmark functional Python code materials property calculations