title: Multimodal Large Language Models for Inverse Molecular Design with
    Retrosynthetic Planning
summary: While large language models (LLMs) have integrated images, adapting them to
  graphs remains challenging, limiting their applications in materials and drug
  design. This difficulty stems from the need for coherent autoregressive
  generation across texts and graphs. To address this, we introduce Llamole, the
  first multimodal LLM capable of interleaved text and graph generation, enabling
  molecular inverse design with retrosynthetic planning. Llamole integrates a
  base LLM with the Graph Diffusion Transformer and Graph Neural Networks for
  multi-conditional molecular generation and reaction inference within texts,
  while the LLM, with enhanced molecular understanding, flexibly controls
  activation among the different graph modules. Additionally, Llamole integrates
  A* search with LLM-based cost functions for efficient retrosynthetic planning.
  We create benchmarking datasets and conduct extensive experiments to evaluate
  Llamole against in-context learning and supervised fine-tuning. Llamole
  significantly outperforms 14 adapted LLMs across 12 metrics for controllable
  molecular design and retrosynthetic planning.
paper_authors._id: 6706008cc21eb22813b18d16
paper_authors.name: Gang Liu
paper_authors._id: 6706008cc21eb22813b18d17
paper_authors.name: Michael Sun
paper_authors._id: 6706008cc21eb22813b18d18
paper_authors.name: Wojciech Matusik
paper_authors._id: 6706008cc21eb22813b18d19
paper_authors.name: Meng Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:43:01.104Z
paper_authors._id: 62cdc931c589e4a9e2411d39
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cdc931c589e4a9e2411d39/Q91Xp6WeI50AjViMzbD3Q.png
paper_authors.fullname: Meng Jiang
paper_authors.type: user
paper_authors.user: mjiang89
paper_authors._id: 6706008cc21eb22813b18d1a
paper_authors.name: Jie Chen
paper_title: Multimodal Large Language Models for Inverse Molecular Design with
    Retrosynthetic Planning
paper_summary: While large language models (LLMs) have integrated images, adapting them to
  graphs remains challenging, limiting their applications in materials and drug
  design. This difficulty stems from the need for coherent autoregressive
  generation across texts and graphs. To address this, we introduce Llamole, the
  first multimodal LLM capable of interleaved text and graph generation, enabling
  molecular inverse design with retrosynthetic planning. Llamole integrates a
  base LLM with the Graph Diffusion Transformer and Graph Neural Networks for
  multi-conditional molecular generation and reaction inference within texts,
  while the LLM, with enhanced molecular understanding, flexibly controls
  activation among the different graph modules. Additionally, Llamole integrates
  A* search with LLM-based cost functions for efficient retrosynthetic planning.
  We create benchmarking datasets and conduct extensive experiments to evaluate
  Llamole against in-context learning and supervised fine-tuning. Llamole
  significantly outperforms 14 adapted LLMs across 12 metrics for controllable
  molecular design and retrosynthetic planning.
paper_ai_keywords: Graph Diffusion Transformer Graph Neural Networks molecular inverse design retrosynthetic planning autoregressive generation multi-conditional molecular generation A* search in-Context learning supervised fine-tuning