Towards 3D Molecule-Text Interpretation in Language Models
Language Models (LMs) have greatly influenced diverse domains. However, their
inherent limitation in comprehending 3D molecular structures has considerably
constrained their potential in the biomolecular domain. To bridge this gap, we
focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular
Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze
3D molecules by equipping the LM with a 3D molecular encoder. This integration
is achieved by a 3D molecule-text projector, bridging the 3D molecular
encoder's representation space and the LM's input space. Moreover, to enhance
3D-MoLM's ability of cross-modal molecular understanding and instruction
following, we meticulously curated a 3D molecule-centric instruction tuning
dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric
instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder
and LM. It significantly surpasses existing baselines on downstream tasks,
including molecule-text retrieval, molecule captioning, and more challenging
open-text molecular QA tasks, especially focusing on 3D-dependent properties.
65f1043f5ec525f2a2d32fe6
Sihang Li
claimed_verified
2024-03-18T09:02:24.810Z
635913a68f38318bfabc90ea
https://cdn-avatars.huggingface.co/v1/production/uploads/635913a68f38318bfabc90ea/29M50hPy2nMwMjMElAEgA.jpeg
Sihang Li
user
Sihangli
65f1043f5ec525f2a2d32fe7
Zhiyuan Liu
claimed_verified
2024-03-19T14:31:26.129Z
6310a3cd531cc21f9e06de6a
https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
Zhiyuan Liu
user
acharkq
65f1043f5ec525f2a2d32fe8
Yanchen Luo
admin_assigned
2024-03-21T11:42:29.191Z
64f04a28f3cd962c21726459
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/MOTc7SWbzc4jdJbMcWMcK.jpeg
LuoYanchen
user
lyc0930
65f1043f5ec525f2a2d32fe9
Xiang Wang
65f1043f5ec525f2a2d32fea
Xiangnan He
65f1043f5ec525f2a2d32feb
Kenji Kawaguchi
65f1043f5ec525f2a2d32fec
Tat-Seng Chua
admin_assigned
2024-03-21T11:43:43.283Z
6570ae84c4993b8fb96f41a8
/avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
Tat-Seng Chua
user
chuats
65f1043f5ec525f2a2d32fed
Qi Tian
Towards 3D Molecule-Text Interpretation in Language Models
Language Models (LMs) have greatly influenced diverse domains. However, their
inherent limitation in comprehending 3D molecular structures has considerably
constrained their potential in the biomolecular domain. To bridge this gap, we
focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular
Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze
3D molecules by equipping the LM with a 3D molecular encoder. This integration
is achieved by a 3D molecule-text projector, bridging the 3D molecular
encoder's representation space and the LM's input space. Moreover, to enhance
3D-MoLM's ability of cross-modal molecular understanding and instruction
following, we meticulously curated a 3D molecule-centric instruction tuning
dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric
instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder
and LM. It significantly surpasses existing baselines on downstream tasks,
including molecule-text retrieval, molecule captioning, and more challenging
open-text molecular QA tasks, especially focusing on 3D-dependent properties.
3D-MoLM enhances language models by integrating a 3D molecular encoder, improving performance on tasks involving 3D molecular structures and instructions.
3D-MoLM 3D molecular language modeling 3D molecular encoder 3D molecule-text projector 3D-MoIT 3D molecule-centric instruction tuning molecule-text retrieval molecule captioning open-text molecular QA