title: Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
summary: We propose fine-tuning large language models for generation of stable
  materials. While unorthodox, fine-tuning large language models on text-encoded
  atomistic data is simple to implement yet reliable, with around 90% of sampled
  structures obeying physical constraints on atom positions and charges. Using
  energy above hull calculations from both learned ML potentials and
  gold-standard DFT calculations, we show that our strongest model (fine-tuned
  LLaMA-2 70B) can generate materials predicted to be metastable at about twice
  the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text
  prompting's inherent flexibility, our models can simultaneously be used for
  unconditional generation of stable material, infilling of partial structures
  and text-conditional generation. Finally, we show that language models' ability
  to capture key symmetries of crystal structures improves with model scale,
  suggesting that the biases of pretrained LLMs are surprisingly well-suited for
  atomistic data.
paper_authors._id: 65c43b4ed2db41f74a93f656
paper_authors.name: Nate Gruver
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:58:18.806Z
paper_authors._id: 645fe8c0236b10b7011d85b2
paper_authors.avatarUrl: /avatars/5d04f719df6ed5cb1d5430d41ef32cf7.svg
paper_authors.fullname: Nate Gruver
paper_authors.type: user
paper_authors.user: ngruver
paper_authors._id: 65c43b4ed2db41f74a93f657
paper_authors.name: Anuroop Sriram
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:58:34.542Z
paper_authors._id: 65022c58b7895899cbc9b223
paper_authors.avatarUrl: /avatars/2a1d7da77890355d4491845b8e54ce97.svg
paper_authors.fullname: anuroop sriram
paper_authors.type: user
paper_authors.user: anuroopmeta
paper_authors._id: 65c43b4ed2db41f74a93f658
paper_authors.name: Andrea Madotto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:58:40.274Z
paper_authors._id: 609bc5719da7ce3ca8930b8e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620821302901-noauth.jpeg
paper_authors.fullname: Andrea Madotto
paper_authors.type: user
paper_authors.user: andreamad8
paper_authors._id: 65c43b4ed2db41f74a93f659
paper_authors.name: Andrew Gordon Wilson
paper_authors._id: 65c43b4ed2db41f74a93f65a
paper_authors.name: C. Lawrence Zitnick
paper_authors._id: 65c43b4ed2db41f74a93f65b
paper_authors.name: Zachary Ulissi
paper_title: Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
paper_summary: We propose fine-tuning large language models for generation of stable
  materials. While unorthodox, fine-tuning large language models on text-encoded
  atomistic data is simple to implement yet reliable, with around 90% of sampled
  structures obeying physical constraints on atom positions and charges. Using
  energy above hull calculations from both learned ML potentials and
  gold-standard DFT calculations, we show that our strongest model (fine-tuned
  LLaMA-2 70B) can generate materials predicted to be metastable at about twice
  the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text
  prompting's inherent flexibility, our models can simultaneously be used for
  unconditional generation of stable material, infilling of partial structures
  and text-conditional generation. Finally, we show that language models' ability
  to capture key symmetries of crystal structures improves with model scale,
  suggesting that the biases of pretrained LLMs are surprisingly well-suited for
  atomistic data.
paper_ai_keywords: fine-tuning large language models text-encoded atomistic data physical constraints energy above hull calculations machine learning (ML) potentials density functional theory (DFT) calculations LLaMA-2 70B CDVAE diffusion model text prompting unconditional generation infilling text-conditional generation crystal structures symmetries pretrained LLMs atomistic data