title: RBench-V: A Primary Assessment for Visual Reasoning Models with
    Multi-modal Outputs
summary: The rapid advancement of native multi-modal models and omni-models,
  exemplified by GPT-4o, Gemini, and o3, with their capability to process and
  generate content across modalities such as text and images, marks a significant
  milestone in the evolution of intelligence. Systematic evaluation of their
  multi-modal output capabilities in visual thinking processes (also known as
  multi-modal chain of thought, M-CoT) becomes critically important. However,
  existing benchmarks for evaluating multi-modal models primarily focus on
  assessing multi-modal inputs and text-only reasoning while neglecting the
  importance of reasoning through multi-modal outputs. In this paper, we present
  a benchmark, dubbed RBench-V, designed to assess models' vision-indispensable
  reasoning abilities. To construct RBench-V, we carefully hand-pick 803
  questions covering math, physics, counting, and games. Unlike previous
  benchmarks that typically specify certain input modalities, RBench-V presents
  problems centered on multi-modal outputs, which require image manipulation such
  as generating novel images and constructing auxiliary lines to support the
  reasoning process. We evaluate numerous open- and closed-source models on
  RBench-V, including o3, Gemini 2.5 Pro, Qwen2.5-VL, etc. Even the
  best-performing model, o3, achieves only 25.8% accuracy on RBench-V, far below
  the human score of 82.3%, highlighting that current models struggle to leverage
  multi-modal reasoning. Data and code are available at
  https://evalmodels.github.io/rbenchv
paper_authors._id: 68308e2697d9a81c8521bc6a
paper_authors.name: Meng-Hao Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:12:57.462Z
paper_authors._id: 62145614b670cb63a38075ba
paper_authors.avatarUrl: /avatars/5e33debde75ae6c87640f63c48c560c6.svg
paper_authors.fullname: MenghaoGuo
paper_authors.type: user
paper_authors.user: MenghaoGuo
paper_authors._id: 68308e2697d9a81c8521bc6b
paper_authors.name: Xuanyu Chu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:12:48.661Z
paper_authors._id: 66b711f9512dac2ac08bc5e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66b711f9512dac2ac08bc5e5/n2kSqNhg-TE56iN_V0xHm.png
paper_authors.fullname: Xuanyu Chu
paper_authors.type: user
paper_authors.user: CXY07
paper_authors._id: 68308e2697d9a81c8521bc6c
paper_authors.name: Qianrui Yang
paper_authors._id: 68308e2697d9a81c8521bc6d
paper_authors.name: Zhe-Han Mo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:12:52.048Z
paper_authors._id: 6816bd8e0499f6c7c7b89601
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6816bd8e0499f6c7c7b89601/-dkIPxjOGbdwDZFxhkBMC.jpeg
paper_authors.fullname: Zhe-Han Mo
paper_authors.type: user
paper_authors.user: Mo-ZheHan
paper_authors._id: 68308e2697d9a81c8521bc6e
paper_authors.name: Yiqing Shen
paper_authors._id: 68308e2697d9a81c8521bc6f
paper_authors.name: Pei-lin Li
paper_authors._id: 68308e2697d9a81c8521bc70
paper_authors.name: Xinjie Lin
paper_authors._id: 68308e2697d9a81c8521bc71
paper_authors.name: Jinnian Zhang
paper_authors._id: 68308e2697d9a81c8521bc72
paper_authors.name: Xin-Sheng Chen
paper_authors._id: 68308e2697d9a81c8521bc73
paper_authors.name: Yi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:12:54.379Z
paper_authors._id: 63b2efb5922f26a27e76381c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b2efb5922f26a27e76381c/zOQAt_xywiY8eTvvQOrmQ.png
paper_authors.fullname: Yi Zhang
paper_authors.type: user
paper_authors.user: uyzhang
paper_authors._id: 68308e2697d9a81c8521bc74
paper_authors.name: Kiyohiro Nakayama
paper_authors._id: 68308e2697d9a81c8521bc75
paper_authors.name: Zhengyang Geng
paper_authors._id: 68308e2697d9a81c8521bc76
paper_authors.name: Houwen Peng
paper_authors._id: 68308e2697d9a81c8521bc77
paper_authors.name: Han Hu
paper_authors._id: 68308e2697d9a81c8521bc78
paper_authors.name: Shi-Nin Hu
paper_title: RBench-V: A Primary Assessment for Visual Reasoning Models with
    Multi-modal Outputs
paper_summary: The rapid advancement of native multi-modal models and omni-models,
  exemplified by GPT-4o, Gemini, and o3, with their capability to process and
  generate content across modalities such as text and images, marks a significant
  milestone in the evolution of intelligence. Systematic evaluation of their
  multi-modal output capabilities in visual thinking processes (also known as
  multi-modal chain of thought, M-CoT) becomes critically important. However,
  existing benchmarks for evaluating multi-modal models primarily focus on
  assessing multi-modal inputs and text-only reasoning while neglecting the
  importance of reasoning through multi-modal outputs. In this paper, we present
  a benchmark, dubbed RBench-V, designed to assess models' vision-indispensable
  reasoning abilities. To construct RBench-V, we carefully hand-pick 803
  questions covering math, physics, counting, and games. Unlike previous
  benchmarks that typically specify certain input modalities, RBench-V presents
  problems centered on multi-modal outputs, which require image manipulation such
  as generating novel images and constructing auxiliary lines to support the
  reasoning process. We evaluate numerous open- and closed-source models on
  RBench-V, including o3, Gemini 2.5 Pro, Qwen2.5-VL, etc. Even the
  best-performing model, o3, achieves only 25.8% accuracy on RBench-V, far below
  the human score of 82.3%, highlighting that current models struggle to leverage
  multi-modal reasoning. Data and code are available at
  https://evalmodels.github.io/rbenchv
paper_ai_summary: A benchmark called RBench-V evaluates multi-modal models' vision-indispensable reasoning through image manipulation and auxiliary line construction, demonstrating that current models struggle with multi-modal outputs.
paper_ai_keywords: multi-modal models omni-models GPT-4o Gemini o3 multi-modal chain of thought M-CoT RBench-V image manipulation auxiliary lines