Can Multimodal Foundation Models Understand Schematic Diagrams? An
  Empirical Study on Information-Seeking QA over Scientific Papers
This paper introduces MISS-QA, the first benchmark specifically designed to
evaluate the ability of models to interpret schematic diagrams within
scientific literature. MISS-QA comprises 1,500 expert-annotated examples over
465 scientific papers. In this benchmark, models are tasked with interpreting
schematic diagrams that illustrate research overviews and answering
corresponding information-seeking questions based on the broader context of the
paper. We assess the performance of 18 frontier multimodal foundation models,
including o4-mini, Gemini-2.5-Flash, and Qwen2.5-VL. We reveal a significant
performance gap between these models and human experts on MISS-QA. Our analysis
of model performance on unanswerable questions and our detailed error analysis
further highlight the strengths and limitations of current models, offering key
insights to enhance models in comprehending multimodal scientific literature.
68771a98257d4f04353707b2
Yilun Zhao
claimed_verified
2025-07-16T08:15:45.442Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
68771a98257d4f04353707b3
Chengye Wang
68771a98257d4f04353707b4
Chuhan Li
68771a98257d4f04353707b5
Arman Cohan
Can Multimodal Foundation Models Understand Schematic Diagrams? An
  Empirical Study on Information-Seeking QA over Scientific Papers
This paper introduces MISS-QA, the first benchmark specifically designed to
evaluate the ability of models to interpret schematic diagrams within
scientific literature. MISS-QA comprises 1,500 expert-annotated examples over
465 scientific papers. In this benchmark, models are tasked with interpreting
schematic diagrams that illustrate research overviews and answering
corresponding information-seeking questions based on the broader context of the
paper. We assess the performance of 18 frontier multimodal foundation models,
including o4-mini, Gemini-2.5-Flash, and Qwen2.5-VL. We reveal a significant
performance gap between these models and human experts on MISS-QA. Our analysis
of model performance on unanswerable questions and our detailed error analysis
further highlight the strengths and limitations of current models, offering key
insights to enhance models in comprehending multimodal scientific literature.
A benchmark evaluates multimodal models' ability to interpret schematic diagrams in scientific literature, revealing performance gaps compared to human experts.
multimodal foundation models o4-mini Gemini-2.5-Flash Qwen2.5-VL schematic diagrams scientific literature information-seeking questions unanswerable questions error analysis