BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning
  Dataset
In this paper, we introduce BMMR, a large-scale bilingual, multimodal,
multi-disciplinary reasoning dataset for the community to develop and evaluate
large multimodal models (LMMs). BMMR comprises 110k college-level questions
spanning 300 UNESCO-defined subjects, spanning diverse formats-multiple-choice,
fill-in-the-blank, and open-ended QA-and sourced from both print and digital
media such as books, exams, and quizzes. All data are curated and filtered via
a human-in-the-loop and scalable framework, and each instance is paired with a
high-quality reasoning path. The dataset is organized into two parts: BMMR-Eval
that comprises 20,458 high-quality instances to comprehensively assess LMMs'
knowledge and reasoning across multiple disciplines in both Chinese and
English; and BMMR-Train that contains 88,991 instances to support further
research and development, extending the current focus on mathematical reasoning
to diverse disciplines and domains. In addition, we propose the process-based
multi-discipline verifier (i.e., BMMR-Verifier) for accurate and fine-grained
evaluation of reasoning paths. Extensive experiments on 24 models reveal that
(i) even SOTA models (e.g., o3 and Gemini-2.5-Pro) leave substantial headroom
on BMMR-Eval; (ii) reasoning models exhibit discipline bias and outperform LMMs
only on specific subjects; (iii) open-source models still trail their
proprietary counterparts; and (iv) fine-tuning on BMMR-Train narrows this gap.
Additionally, we conduct reasoning-chain analyses using BMMR-Verifier and other
in-depth studies, uncovering the challenges LMMs currently face in
multidisciplinary reasoning. We will release the data, and we hope our work can
offer insights and contributions to the community.
686c90d4364e2ad167eb53d8
Zhiheng Xi
686c90d4364e2ad167eb53d9
Guanyu Li
686c90d4364e2ad167eb53da
Yutao Fan
686c90d4364e2ad167eb53db
Honglin Guo
686c90d4364e2ad167eb53dc
Yufang Liu
686c90d4364e2ad167eb53dd
Xiaoran Fan
686c90d4364e2ad167eb53de
Jiaqi Liu
686c90d4364e2ad167eb53df
Jingchao Ding
686c90d4364e2ad167eb53e0
Wangmeng Zuo
686c90d4364e2ad167eb53e1
Zhenfei Yin
686c90d4364e2ad167eb53e2
Lei Bai
686c90d4364e2ad167eb53e3
Tao Ji
686c90d4364e2ad167eb53e4
Tao Gui
686c90d4364e2ad167eb53e5
Qi Zhang
686c90d4364e2ad167eb53e6
Xuanjing Huang
BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning
  Dataset
In this paper, we introduce BMMR, a large-scale bilingual, multimodal,
multi-disciplinary reasoning dataset for the community to develop and evaluate
large multimodal models (LMMs). BMMR comprises 110k college-level questions
spanning 300 UNESCO-defined subjects, spanning diverse formats-multiple-choice,
fill-in-the-blank, and open-ended QA-and sourced from both print and digital
media such as books, exams, and quizzes. All data are curated and filtered via
a human-in-the-loop and scalable framework, and each instance is paired with a
high-quality reasoning path. The dataset is organized into two parts: BMMR-Eval
that comprises 20,458 high-quality instances to comprehensively assess LMMs'
knowledge and reasoning across multiple disciplines in both Chinese and
English; and BMMR-Train that contains 88,991 instances to support further
research and development, extending the current focus on mathematical reasoning
to diverse disciplines and domains. In addition, we propose the process-based
multi-discipline verifier (i.e., BMMR-Verifier) for accurate and fine-grained
evaluation of reasoning paths. Extensive experiments on 24 models reveal that
(i) even SOTA models (e.g., o3 and Gemini-2.5-Pro) leave substantial headroom
on BMMR-Eval; (ii) reasoning models exhibit discipline bias and outperform LMMs
only on specific subjects; (iii) open-source models still trail their
proprietary counterparts; and (iv) fine-tuning on BMMR-Train narrows this gap.
Additionally, we conduct reasoning-chain analyses using BMMR-Verifier and other
in-depth studies, uncovering the challenges LMMs currently face in
multidisciplinary reasoning. We will release the data, and we hope our work can
offer insights and contributions to the community.
A large-scale bilingual, multimodal, multi-disciplinary reasoning dataset (BMMR) is introduced to evaluate and develop large multimodal models (LMMs) across various disciplines and formats, with a focus on reasoning paths and discipline-specific performance.
bilingual multimodal multi-disciplinary reasoning large multimodal models LMMs UNESCO-defined subjects multiple-choice fill-in-the-blank open-ended QA human-in-the-loop scalable framework reasoning path BMMR-Eval BMMR-Train process-based multi-discipline verifier BMMR-Verifier reasoning-chain analyses