title: Towards 3D Molecule-Text Interpretation in Language Models
summary: Language Models (LMs) have greatly influenced diverse domains. However, their
  inherent limitation in comprehending 3D molecular structures has considerably
  constrained their potential in the biomolecular domain. To bridge this gap, we
  focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular
  Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze
  3D molecules by equipping the LM with a 3D molecular encoder. This integration
  is achieved by a 3D molecule-text projector, bridging the 3D molecular
  encoder's representation space and the LM's input space. Moreover, to enhance
  3D-MoLM's ability of cross-modal molecular understanding and instruction
  following, we meticulously curated a 3D molecule-centric instruction tuning
  dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric
  instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder
  and LM. It significantly surpasses existing baselines on downstream tasks,
  including molecule-text retrieval, molecule captioning, and more challenging
  open-text molecular QA tasks, especially focusing on 3D-dependent properties.
paper_authors._id: 65f1043f5ec525f2a2d32fe6
paper_authors.name: Sihang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-18T09:02:24.810Z
paper_authors._id: 635913a68f38318bfabc90ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635913a68f38318bfabc90ea/29M50hPy2nMwMjMElAEgA.jpeg
paper_authors.fullname: Sihang Li
paper_authors.type: user
paper_authors.user: Sihangli
paper_authors._id: 65f1043f5ec525f2a2d32fe7
paper_authors.name: Zhiyuan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-19T14:31:26.129Z
paper_authors._id: 6310a3cd531cc21f9e06de6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: acharkq
paper_authors._id: 65f1043f5ec525f2a2d32fe8
paper_authors.name: Yanchen Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T11:42:29.191Z
paper_authors._id: 64f04a28f3cd962c21726459
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/MOTc7SWbzc4jdJbMcWMcK.jpeg
paper_authors.fullname: LuoYanchen
paper_authors.type: user
paper_authors.user: lyc0930
paper_authors._id: 65f1043f5ec525f2a2d32fe9
paper_authors.name: Xiang Wang
paper_authors._id: 65f1043f5ec525f2a2d32fea
paper_authors.name: Xiangnan He
paper_authors._id: 65f1043f5ec525f2a2d32feb
paper_authors.name: Kenji Kawaguchi
paper_authors._id: 65f1043f5ec525f2a2d32fec
paper_authors.name: Tat-Seng Chua
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T11:43:43.283Z
paper_authors._id: 6570ae84c4993b8fb96f41a8
paper_authors.avatarUrl: /avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
paper_authors.fullname: Tat-Seng Chua
paper_authors.type: user
paper_authors.user: chuats
paper_authors._id: 65f1043f5ec525f2a2d32fed
paper_authors.name: Qi Tian
paper_title: Towards 3D Molecule-Text Interpretation in Language Models
paper_summary: Language Models (LMs) have greatly influenced diverse domains. However, their
  inherent limitation in comprehending 3D molecular structures has considerably
  constrained their potential in the biomolecular domain. To bridge this gap, we
  focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular
  Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze
  3D molecules by equipping the LM with a 3D molecular encoder. This integration
  is achieved by a 3D molecule-text projector, bridging the 3D molecular
  encoder's representation space and the LM's input space. Moreover, to enhance
  3D-MoLM's ability of cross-modal molecular understanding and instruction
  following, we meticulously curated a 3D molecule-centric instruction tuning
  dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric
  instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder
  and LM. It significantly surpasses existing baselines on downstream tasks,
  including molecule-text retrieval, molecule captioning, and more challenging
  open-text molecular QA tasks, especially focusing on 3D-dependent properties.
paper_ai_keywords: 3D-MoLM 3D molecular encoder 3D molecule-text projector 3D molecule-centric instruction tuning 3D-MoIT 3D molecule-text alignment molecule-text retrieval molecule captioning open-text molecular QA