title: Seeing and Understanding: Bridging Vision with Chemical Knowledge Via
    ChemVLM
summary: In this technical report, we propose ChemVLM, the first open-source
  multimodal large language model dedicated to the fields of chemistry, designed
  to address the incompatibility between chemical image understanding and text
  analysis. Built upon the VIT-MLP-LLM architecture, we leverage ChemLLM-20B as
  the foundational large model, endowing our model with robust capabilities in
  understanding and utilizing chemical text knowledge. Additionally, we employ
  InternVIT-6B as a powerful image encoder. We have curated high-quality data
  from the chemical domain, including molecules, reaction formulas, and chemistry
  examination data, and compiled these into a bilingual multimodal
  question-answering dataset. We test the performance of our model on multiple
  open-source benchmarks and three custom evaluation sets. Experimental results
  demonstrate that our model achieves excellent performance, securing
  state-of-the-art results in five out of six involved tasks. Our model can be
  found at https://huggingface.co/AI4Chem/ChemVLM-26B.
paper_authors._id: 66bd4adb6f39f76a395d78ca
paper_authors.name: Junxian Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-15T10:09:21.334Z
paper_authors._id: 656ae4088fb1ddf0d5ec9ac5
paper_authors.avatarUrl: /avatars/e38468d2c0274f3c0f5732f30a2e3436.svg
paper_authors.fullname: Junxian Li
paper_authors.type: user
paper_authors.user: Duke-de-Artois
paper_authors._id: 66bd4adb6f39f76a395d78cb
paper_authors.name: Di Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-28T07:45:07.073Z
paper_authors._id: 64bce15bafd1e46c5504ad38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/bQFX1iFbXEBXcQvUNL811.png
paper_authors.fullname: Di Zhang
paper_authors.type: user
paper_authors.user: di-zhang-fdu
paper_authors._id: 66bd4adb6f39f76a395d78cc
paper_authors.name: Xunzhi Wang
paper_authors._id: 66bd4adb6f39f76a395d78cd
paper_authors.name: Zeying Hao
paper_authors._id: 66bd4adb6f39f76a395d78ce
paper_authors.name: Jingdi Lei
paper_authors._id: 66bd4adb6f39f76a395d78cf
paper_authors.name: Qian Tan
paper_authors._id: 66bd4adb6f39f76a395d78d0
paper_authors.name: Cai Zhou
paper_authors._id: 66bd4adb6f39f76a395d78d1
paper_authors.name: Wei Liu
paper_authors._id: 66bd4adb6f39f76a395d78d2
paper_authors.name: Weiyun Wang
paper_authors._id: 66bd4adb6f39f76a395d78d3
paper_authors.name: Zhe Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:48:58.339Z
paper_authors._id: 64119264f0f81eb569e0d569
paper_authors.avatarUrl: /avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
paper_authors.fullname: Zhe Chen
paper_authors.type: user
paper_authors.user: czczup
paper_authors._id: 66bd4adb6f39f76a395d78d4
paper_authors.name: Wenhai Wang
paper_authors._id: 66bd4adb6f39f76a395d78d5
paper_authors.name: Wei Li
paper_authors._id: 66bd4adb6f39f76a395d78d6
paper_authors.name: Shufei Zhang
paper_authors._id: 66bd4adb6f39f76a395d78d7
paper_authors.name: Mao Su
paper_authors._id: 66bd4adb6f39f76a395d78d8
paper_authors.name: Wanli Ouyang
paper_authors._id: 66bd4adb6f39f76a395d78d9
paper_authors.name: Yuqiang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-15T10:39:09.855Z
paper_authors._id: 65cb61ba0390fce629de99d1
paper_authors.avatarUrl: /avatars/9b4b4556ffd8de215dc37b02366d781b.svg
paper_authors.fullname: Yuqiang Li
paper_authors.type: user
paper_authors.user: yuqiangli
paper_authors._id: 66bd4adb6f39f76a395d78da
paper_authors.name: Dongzhan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-15T10:38:44.611Z
paper_authors._id: 6538b861613fe158bd581e35
paper_authors.avatarUrl: /avatars/6817dbfe903675721fd227058b0a91ac.svg
paper_authors.fullname: Dongzhan Zhou
paper_authors.type: user
paper_authors.user: schrodingers-tiger
paper_title: Seeing and Understanding: Bridging Vision with Chemical Knowledge Via
    ChemVLM
paper_summary: In this technical report, we propose ChemVLM, the first open-source
  multimodal large language model dedicated to the fields of chemistry, designed
  to address the incompatibility between chemical image understanding and text
  analysis. Built upon the VIT-MLP-LLM architecture, we leverage ChemLLM-20B as
  the foundational large model, endowing our model with robust capabilities in
  understanding and utilizing chemical text knowledge. Additionally, we employ
  InternVIT-6B as a powerful image encoder. We have curated high-quality data
  from the chemical domain, including molecules, reaction formulas, and chemistry
  examination data, and compiled these into a bilingual multimodal
  question-answering dataset. We test the performance of our model on multiple
  open-source benchmarks and three custom evaluation sets. Experimental results
  demonstrate that our model achieves excellent performance, securing
  state-of-the-art results in five out of six involved tasks. Our model can be
  found at https://huggingface.co/AI4Chem/ChemVLM-26B.
paper_ai_keywords: ChemVLM VIT-MLP-LLM architecture ChemLLM-20B InternVIT-6B