VLM-R^3: Region Recognition, Reasoning, and Refinement for Enhanced
  Multimodal Chain-of-Thought
Recently, reasoning-based MLLMs have achieved a degree of success in
generating long-form textual reasoning chains. However, they still struggle
with complex tasks that necessitate dynamic and iterative focusing on and
revisiting of visual regions to achieve precise grounding of textual reasoning
in visual evidence. We introduce VLM-R^3 (Visual
Language Model with Region Recognition and
Reasoning), a framework that equips an MLLM with the ability to (i)
decide when additional visual evidence is needed, (ii) determine
where to ground within the image, and (iii) seamlessly weave the
relevant sub-image content back into an interleaved chain-of-thought. The core
of our method is Region-Conditioned Reinforcement Policy Optimization
(R-GRPO), a training paradigm that rewards the model for selecting informative
regions, formulating appropriate transformations (e.g.\ crop, zoom), and
integrating the resulting visual context into subsequent reasoning steps. To
bootstrap this policy, we compile a modest but carefully curated Visuo-Lingual
Interleaved Rationale (VLIR) corpus that provides step-level supervision on
region selection and textual justification. Extensive experiments on MathVista,
ScienceQA, and other benchmarks show that VLM-R^3 sets a new state of the art
in zero-shot and few-shot settings, with the largest gains appearing on
questions demanding subtle spatial reasoning or fine-grained visual cue
extraction.
68302db3b73ef22aebdce9c2
Chaoya Jiang
68302db3b73ef22aebdce9c3
Yongrui Heng
68302db3b73ef22aebdce9c4
Wei Ye
68302db3b73ef22aebdce9c5
Han Yang
68302db3b73ef22aebdce9c6
Haiyang Xu
claimed_verified
2025-05-26T08:14:45.417Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
68302db3b73ef22aebdce9c7
Ming Yan
68302db3b73ef22aebdce9c8
Ji Zhang
68302db3b73ef22aebdce9c9
Fei Huang
68302db3b73ef22aebdce9ca
Shikun Zhang
VLM-R^3: Region Recognition, Reasoning, and Refinement for Enhanced
  Multimodal Chain-of-Thought
Recently, reasoning-based MLLMs have achieved a degree of success in
generating long-form textual reasoning chains. However, they still struggle
with complex tasks that necessitate dynamic and iterative focusing on and
revisiting of visual regions to achieve precise grounding of textual reasoning
in visual evidence. We introduce VLM-R^3 (Visual
Language Model with Region Recognition and
Reasoning), a framework that equips an MLLM with the ability to (i)
decide when additional visual evidence is needed, (ii) determine
where to ground within the image, and (iii) seamlessly weave the
relevant sub-image content back into an interleaved chain-of-thought. The core
of our method is Region-Conditioned Reinforcement Policy Optimization
(R-GRPO), a training paradigm that rewards the model for selecting informative
regions, formulating appropriate transformations (e.g.\ crop, zoom), and
integrating the resulting visual context into subsequent reasoning steps. To
bootstrap this policy, we compile a modest but carefully curated Visuo-Lingual
Interleaved Rationale (VLIR) corpus that provides step-level supervision on
region selection and textual justification. Extensive experiments on MathVista,
ScienceQA, and other benchmarks show that VLM-R^3 sets a new state of the art
in zero-shot and few-shot settings, with the largest gains appearing on
questions demanding subtle spatial reasoning or fine-grained visual cue
extraction.
VLM-R3 enhances multi-modal language models with region recognition and reasoning, achieving state-of-the-art performance on visual question answering tasks through region-conditioned reinforcement policy optimization.
VLM-R3 Region-Conditioned Reinforcement Policy Optimization (R-GRPO) Visuo-Lingual Interleaved Rationale (VLIR) corpus MathVista ScienceQA