Seeing and Understanding: Bridging Vision with Chemical Knowledge Via
  ChemVLM
In this technical report, we propose ChemVLM, the first open-source
multimodal large language model dedicated to the fields of chemistry, designed
to address the incompatibility between chemical image understanding and text
analysis. Built upon the VIT-MLP-LLM architecture, we leverage ChemLLM-20B as
the foundational large model, endowing our model with robust capabilities in
understanding and utilizing chemical text knowledge. Additionally, we employ
InternVIT-6B as a powerful image encoder. We have curated high-quality data
from the chemical domain, including molecules, reaction formulas, and chemistry
examination data, and compiled these into a bilingual multimodal
question-answering dataset. We test the performance of our model on multiple
open-source benchmarks and three custom evaluation sets. Experimental results
demonstrate that our model achieves excellent performance, securing
state-of-the-art results in five out of six involved tasks. Our model can be
found at https://huggingface.co/AI4Chem/ChemVLM-26B.
66bd4adb6f39f76a395d78ca
Junxian Li
claimed_verified
2024-08-15T10:09:21.334Z
656ae4088fb1ddf0d5ec9ac5
/avatars/e38468d2c0274f3c0f5732f30a2e3436.svg
Junxian Li
user
Duke-de-Artois
66bd4adb6f39f76a395d78cb
Di Zhang
extracted_confirmed
2024-11-28T07:45:07.073Z
64bce15bafd1e46c5504ad38
https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/vkEjiu-mIagKlrXzDH75o.png
Di Zhang
user
di-zhang-fdu
66bd4adb6f39f76a395d78cc
Xunzhi Wang
66bd4adb6f39f76a395d78cd
Zeying Hao
66bd4adb6f39f76a395d78ce
Jingdi Lei
66bd4adb6f39f76a395d78cf
Qian Tan
66bd4adb6f39f76a395d78d0
Cai Zhou
66bd4adb6f39f76a395d78d1
Wei Liu
66bd4adb6f39f76a395d78d2
Weiyun Wang
66bd4adb6f39f76a395d78d3
Zhe Chen
claimed_verified
2024-12-09T14:48:58.339Z
64119264f0f81eb569e0d569
/avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
Zhe Chen
user
czczup
66bd4adb6f39f76a395d78d4
Wenhai Wang
66bd4adb6f39f76a395d78d5
Wei Li
66bd4adb6f39f76a395d78d6
Shufei Zhang
66bd4adb6f39f76a395d78d7
Mao Su
66bd4adb6f39f76a395d78d8
Wanli Ouyang
66bd4adb6f39f76a395d78d9
Yuqiang Li
admin_assigned
2024-08-15T10:39:09.855Z
65cb61ba0390fce629de99d1
/avatars/9b4b4556ffd8de215dc37b02366d781b.svg
Yuqiang Li
user
yuqiangli
66bd4adb6f39f76a395d78da
Dongzhan Zhou
admin_assigned
2024-08-15T10:38:44.611Z
6538b861613fe158bd581e35
/avatars/6817dbfe903675721fd227058b0a91ac.svg
Dongzhan Zhou
user
schrodingers-tiger
Seeing and Understanding: Bridging Vision with Chemical Knowledge Via
  ChemVLM
In this technical report, we propose ChemVLM, the first open-source
multimodal large language model dedicated to the fields of chemistry, designed
to address the incompatibility between chemical image understanding and text
analysis. Built upon the VIT-MLP-LLM architecture, we leverage ChemLLM-20B as
the foundational large model, endowing our model with robust capabilities in
understanding and utilizing chemical text knowledge. Additionally, we employ
InternVIT-6B as a powerful image encoder. We have curated high-quality data
from the chemical domain, including molecules, reaction formulas, and chemistry
examination data, and compiled these into a bilingual multimodal
question-answering dataset. We test the performance of our model on multiple
open-source benchmarks and three custom evaluation sets. Experimental results
demonstrate that our model achieves excellent performance, securing
state-of-the-art results in five out of six involved tasks. Our model can be
found at https://huggingface.co/AI4Chem/ChemVLM-26B.
ChemVLM, a multimodal language model combiningtext and image understanding in chemistry, achieves state-of-the-art performance using VIT-MLP-LLM, ChemLLM-20B, and InternVIT-6B architectures.
VIT-MLP-LLM ChemLLM-20B InternVIT-6B multimodal language model chemical text knowledge image encoder bilingual multimodal question-answering dataset