Multimodal Large Language Models for Inverse Molecular Design with
  Retrosynthetic Planning
While large language models (LLMs) have integrated images, adapting them to
graphs remains challenging, limiting their applications in materials and drug
design. This difficulty stems from the need for coherent autoregressive
generation across texts and graphs. To address this, we introduce Llamole, the
first multimodal LLM capable of interleaved text and graph generation, enabling
molecular inverse design with retrosynthetic planning. Llamole integrates a
base LLM with the Graph Diffusion Transformer and Graph Neural Networks for
multi-conditional molecular generation and reaction inference within texts,
while the LLM, with enhanced molecular understanding, flexibly controls
activation among the different graph modules. Additionally, Llamole integrates
A* search with LLM-based cost functions for efficient retrosynthetic planning.
We create benchmarking datasets and conduct extensive experiments to evaluate
Llamole against in-context learning and supervised fine-tuning. Llamole
significantly outperforms 14 adapted LLMs across 12 metrics for controllable
molecular design and retrosynthetic planning.
6706008cc21eb22813b18d16
Gang Liu
6706008cc21eb22813b18d17
Michael Sun
6706008cc21eb22813b18d18
Wojciech Matusik
6706008cc21eb22813b18d19
Meng Jiang
admin_assigned
2024-10-10T09:43:01.104Z
62cdc931c589e4a9e2411d39
https://cdn-avatars.huggingface.co/v1/production/uploads/62cdc931c589e4a9e2411d39/Q91Xp6WeI50AjViMzbD3Q.png
Meng Jiang
user
mjiang89
6706008cc21eb22813b18d1a
Jie Chen
Multimodal Large Language Models for Inverse Molecular Design with
  Retrosynthetic Planning
While large language models (LLMs) have integrated images, adapting them to
graphs remains challenging, limiting their applications in materials and drug
design. This difficulty stems from the need for coherent autoregressive
generation across texts and graphs. To address this, we introduce Llamole, the
first multimodal LLM capable of interleaved text and graph generation, enabling
molecular inverse design with retrosynthetic planning. Llamole integrates a
base LLM with the Graph Diffusion Transformer and Graph Neural Networks for
multi-conditional molecular generation and reaction inference within texts,
while the LLM, with enhanced molecular understanding, flexibly controls
activation among the different graph modules. Additionally, Llamole integrates
A* search with LLM-based cost functions for efficient retrosynthetic planning.
We create benchmarking datasets and conduct extensive experiments to evaluate
Llamole against in-context learning and supervised fine-tuning. Llamole
significantly outperforms 14 adapted LLMs across 12 metrics for controllable
molecular design and retrosynthetic planning.
Llamole, a multimodal LLM integrating Graph Diffusion Transformer and Graph Neural Networks, excels in molecular inverse design and retrosynthetic planning by interleaving text and graph generation.
Llamole Graph Diffusion Transformer Graph Neural Networks multimodal LLM molecular inverse design retrosynthetic planning in-context learning supervised fine-tuning