High-Resolution Visual Reasoning via Multi-Turn Grounding-Based
  Reinforcement Learning
State-of-the-art large multi-modal models (LMMs) face challenges when
processing high-resolution images, as these inputs are converted into enormous
visual tokens, many of which are irrelevant to the downstream task. In this
paper, we propose Multi-turn Grounding-based Policy Optimization (MGPO), an
end-to-end reinforcement learning (RL) framework that enables LMMs to
iteratively focus on key visual regions by automatically cropping sub-images,
based on model-predicted grounding coordinates within a multi-turn conversation
framework. Compared to supervised fine-tuning (SFT), which requires costly
additional grounding annotations, our approach highlights that LMMs can emerge
robust grounding abilities during the RL training process, leveraging only a
binary reward function derived from the correctness of the final answer.
Additionally, we observe that LMMs struggle to autonomously trigger visual
grounding during the rollout process. To address this cold start problem, we
design a multi-turn conversational template and restrict policy loss
computation to model outputs generated across multiple dialogue rounds, thereby
promoting stable optimization. Extensive experiments demonstrate that, when
trained on standard visual-question-short answering data without grounding
annotations, MGPO effectively elicits stronger grounding capabilities compared
to GRPO, leading to 5.4\% improvement on in-distribution MME-Realworld and
5.2\% improvement on the challenging out-of-distribution (OOD) V* Bench.
Notably, MGPO post-training on Qwen2.5-VL-7B with 21K samples surpasses
OpenAI's o1 and GPT-4o models on the OOD V* Bench. Codes are available at
https://github.com/EvolvingLMMs-Lab/MGPO.
686deb33cb5725779c60b49c
Xinyu Huang
claimed_verified
2025-07-11T08:03:47.084Z
63bf7ba8da08ed0544ff20e9
https://cdn-avatars.huggingface.co/v1/production/uploads/1673493776367-63bf7ba8da08ed0544ff20e9.jpeg
Xinyu Huang
user
xinyu1205
686deb33cb5725779c60b49d
Yuhao Dong
claimed_verified
2025-07-10T07:10:02.844Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
user
THUdyh
686deb33cb5725779c60b49e
Weiwei Tian
686deb33cb5725779c60b49f
Bo Li
686deb33cb5725779c60b4a0
Rui Feng
686deb33cb5725779c60b4a1
Ziwei Liu
High-Resolution Visual Reasoning via Multi-Turn Grounding-Based
  Reinforcement Learning
State-of-the-art large multi-modal models (LMMs) face challenges when
processing high-resolution images, as these inputs are converted into enormous
visual tokens, many of which are irrelevant to the downstream task. In this
paper, we propose Multi-turn Grounding-based Policy Optimization (MGPO), an
end-to-end reinforcement learning (RL) framework that enables LMMs to
iteratively focus on key visual regions by automatically cropping sub-images,
based on model-predicted grounding coordinates within a multi-turn conversation
framework. Compared to supervised fine-tuning (SFT), which requires costly
additional grounding annotations, our approach highlights that LMMs can emerge
robust grounding abilities during the RL training process, leveraging only a
binary reward function derived from the correctness of the final answer.
Additionally, we observe that LMMs struggle to autonomously trigger visual
grounding during the rollout process. To address this cold start problem, we
design a multi-turn conversational template and restrict policy loss
computation to model outputs generated across multiple dialogue rounds, thereby
promoting stable optimization. Extensive experiments demonstrate that, when
trained on standard visual-question-short answering data without grounding
annotations, MGPO effectively elicits stronger grounding capabilities compared
to GRPO, leading to 5.4\% improvement on in-distribution MME-Realworld and
5.2\% improvement on the challenging out-of-distribution (OOD) V* Bench.
Notably, MGPO post-training on Qwen2.5-VL-7B with 21K samples surpasses
OpenAI's o1 and GPT-4o models on the OOD V* Bench. Codes are available at
https://github.com/EvolvingLMMs-Lab/MGPO.
MGPO, an end-to-end reinforcement learning framework, enhances large multi-modal models' visual grounding abilities through iterative sub-image cropping, improving performance on visual-question answering tasks without additional grounding annotations.
multi-modal models reinforcement learning visual tokens grounding coordinates multi-turn conversation supervised fine-tuning binary reward function visual grounding policy loss conversational template MME-Realworld V* Bench Qwen2.5-VL-7B OpenAI's o1 GPT-4o