GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular
  Structure Recognition
Optical Chemical Structure Recognition (OCSR) is crucial for digitizing
chemical knowledge by converting molecular images into machine-readable
formats. While recent vision-language models (VLMs) have shown potential in
this task, their image-captioning approach often struggles with complex
molecular structures and inconsistent annotations. To overcome these
challenges, we introduce GTR-Mol-VLM, a novel framework featuring two key
innovations: (1) the Graph Traversal as Visual Chain of Thought
mechanism that emulates human reasoning by incrementally parsing molecular
graphs through sequential atom-bond predictions, and (2) the data-centric
principle of Faithfully Recognize What You've Seen, which addresses
the mismatch between abbreviated structures in images and their expanded
annotations. To support model development, we constructed GTR-CoT-1.3M, a
large-scale instruction-tuning dataset with meticulously corrected annotations,
and introduced MolRec-Bench, the first benchmark designed for a fine-grained
evaluation of graph-parsing accuracy in OCSR. Comprehensive experiments
demonstrate that GTR-Mol-VLM achieves superior results compared to specialist
models, chemistry-domain VLMs, and commercial general-purpose VLMs. Notably, in
scenarios involving molecular images with functional group abbreviations,
GTR-Mol-VLM outperforms the second-best baseline by approximately 14 percentage
points, both in SMILES-based and graph-based metrics. We hope that this work
will drive OCSR technology to more effectively meet real-world needs, thereby
advancing the fields of cheminformatics and AI for Science. We will release
GTR-CoT at https://github.com/opendatalab/GTR-CoT.
684794a43ec10bdd8ab4de24
Jingchao Wang
claimed_verified
2025-06-10T08:44:15.857Z
65eaa07cb6c760d77468b4b6
/avatars/4a1aae58986b40444351e0a167ca807c.svg
Jingchao Wang
user
jcwang0602
684794a43ec10bdd8ab4de25
Haote Yang
claimed_verified
2025-06-10T08:44:18.060Z
65fd45473ccf43503350d837
https://cdn-avatars.huggingface.co/v1/production/uploads/65fd45473ccf43503350d837/i8r1uUpD0dP1vrrWucO6E.jpeg
Haote Yang
user
Hoter
684794a43ec10bdd8ab4de26
Jiang Wu
684794a43ec10bdd8ab4de27
Yifan He
684794a43ec10bdd8ab4de28
Xingjian Wei
684794a43ec10bdd8ab4de29
Yinfan Wang
684794a43ec10bdd8ab4de2a
Chengjin Liu
684794a43ec10bdd8ab4de2b
Lingli Ge
684794a43ec10bdd8ab4de2c
Lijun Wu
684794a43ec10bdd8ab4de2d
Bin Wang
684794a43ec10bdd8ab4de2e
Dahua Lin
684794a43ec10bdd8ab4de2f
Conghui He
GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular
  Structure Recognition
Optical Chemical Structure Recognition (OCSR) is crucial for digitizing
chemical knowledge by converting molecular images into machine-readable
formats. While recent vision-language models (VLMs) have shown potential in
this task, their image-captioning approach often struggles with complex
molecular structures and inconsistent annotations. To overcome these
challenges, we introduce GTR-Mol-VLM, a novel framework featuring two key
innovations: (1) the Graph Traversal as Visual Chain of Thought
mechanism that emulates human reasoning by incrementally parsing molecular
graphs through sequential atom-bond predictions, and (2) the data-centric
principle of Faithfully Recognize What You've Seen, which addresses
the mismatch between abbreviated structures in images and their expanded
annotations. To support model development, we constructed GTR-CoT-1.3M, a
large-scale instruction-tuning dataset with meticulously corrected annotations,
and introduced MolRec-Bench, the first benchmark designed for a fine-grained
evaluation of graph-parsing accuracy in OCSR. Comprehensive experiments
demonstrate that GTR-Mol-VLM achieves superior results compared to specialist
models, chemistry-domain VLMs, and commercial general-purpose VLMs. Notably, in
scenarios involving molecular images with functional group abbreviations,
GTR-Mol-VLM outperforms the second-best baseline by approximately 14 percentage
points, both in SMILES-based and graph-based metrics. We hope that this work
will drive OCSR technology to more effectively meet real-world needs, thereby
advancing the fields of cheminformatics and AI for Science. We will release
GTR-CoT at https://github.com/opendatalab/GTR-CoT.
GTR-Mol-VLM, featuring graph traversal and data-centric principles, outperforms existing models in Optical Chemical Structure Recognition by accurately parsing molecular graphs and handling abbreviated structures.
Graph Traversal as Visual Chain of Thought Faithfully Recognize What You've Seen GTR-CoT-1.3M MolRec-Bench graph-parsing accuracy Optical Chemical Structure Recognition VLMs SMILES-based graph-based metrics