Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
We propose fine-tuning large language models for generation of stable
materials. While unorthodox, fine-tuning large language models on text-encoded
atomistic data is simple to implement yet reliable, with around 90% of sampled
structures obeying physical constraints on atom positions and charges. Using
energy above hull calculations from both learned ML potentials and
gold-standard DFT calculations, we show that our strongest model (fine-tuned
LLaMA-2 70B) can generate materials predicted to be metastable at about twice
the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text
prompting's inherent flexibility, our models can simultaneously be used for
unconditional generation of stable material, infilling of partial structures
and text-conditional generation. Finally, we show that language models' ability
to capture key symmetries of crystal structures improves with model scale,
suggesting that the biases of pretrained LLMs are surprisingly well-suited for
atomistic data.
65c43b4ed2db41f74a93f656
Nate Gruver
admin_assigned
2024-02-08T11:58:18.806Z
645fe8c0236b10b7011d85b2
/avatars/5d04f719df6ed5cb1d5430d41ef32cf7.svg
Nate Gruver
user
ngruver
65c43b4ed2db41f74a93f657
Anuroop Sriram
admin_assigned
2024-02-08T11:58:34.542Z
65022c58b7895899cbc9b223
/avatars/2a1d7da77890355d4491845b8e54ce97.svg
anuroop sriram
user
anuroopmeta
65c43b4ed2db41f74a93f658
Andrea Madotto
admin_assigned
2024-02-08T11:58:40.274Z
609bc5719da7ce3ca8930b8e
https://cdn-avatars.huggingface.co/v1/production/uploads/1620821302901-noauth.jpeg
Andrea Madotto
user
andreamad8
65c43b4ed2db41f74a93f659
Andrew Gordon Wilson
65c43b4ed2db41f74a93f65a
C. Lawrence Zitnick
65c43b4ed2db41f74a93f65b
Zachary Ulissi
Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
We propose fine-tuning large language models for generation of stable
materials. While unorthodox, fine-tuning large language models on text-encoded
atomistic data is simple to implement yet reliable, with around 90% of sampled
structures obeying physical constraints on atom positions and charges. Using
energy above hull calculations from both learned ML potentials and
gold-standard DFT calculations, we show that our strongest model (fine-tuned
LLaMA-2 70B) can generate materials predicted to be metastable at about twice
the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text
prompting's inherent flexibility, our models can simultaneously be used for
unconditional generation of stable material, infilling of partial structures
and text-conditional generation. Finally, we show that language models' ability
to capture key symmetries of crystal structures improves with model scale,
suggesting that the biases of pretrained LLMs are surprisingly well-suited for
atomistic data.
Fine-tuning large language models on atomistic data for material generation demonstrates high constraint adherence and competitive metastable material production compared to diffusion models, leveraging text prompting flexibility and scale-dependent symmetry capture.
fine-tuning large language models text-encoded atomistic data energy above hull ML potentials gold-standard DFT calculations LLaMA-2 CDVAE diffusion model text prompting unconditional generation infilling symmetries crystal structures pretrained LLMs