UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement
  Learning
Traditional visual grounding methods primarily focus on single-image
scenarios with simple textual references. However, extending these methods to
real-world scenarios that involve implicit and complex instructions,
particularly in conjunction with multiple images, poses significant challenges,
which is mainly due to the lack of advanced reasoning ability across diverse
multi-modal contexts. In this work, we aim to address the more practical
universal grounding task, and propose UniVG-R1, a reasoning guided multimodal
large language model (MLLM) for universal visual grounding, which enhances
reasoning capabilities through reinforcement learning (RL) combined with
cold-start data. Specifically, we first construct a high-quality
Chain-of-Thought (CoT) grounding dataset, annotated with detailed reasoning
chains, to guide the model towards correct reasoning paths via supervised
fine-tuning. Subsequently, we perform rule-based reinforcement learning to
encourage the model to identify correct reasoning chains, thereby incentivizing
its reasoning capabilities. In addition, we identify a difficulty bias arising
from the prevalence of easy samples as RL training progresses, and we propose a
difficulty-aware weight adjustment strategy to further strengthen the
performance. Experimental results demonstrate the effectiveness of UniVG-R1,
which achieves state-of-the-art performance on MIG-Bench with a 9.1%
improvement over the previous method. Furthermore, our model exhibits strong
generalizability, achieving an average improvement of 23.4% in zero-shot
performance across four image and video reasoning grounding benchmarks. The
project page can be accessed at https://amap-ml.github.io/UniVG-R1-page/.
682db25d265177367e35d5b1
Sule Bai
claimed_verified
2025-05-28T09:01:23.476Z
682d81e389b5f79a1d7b9682
/avatars/4e0b4f1654de8c74da5b54b154de3e65.svg
SuleBai
user
SuleBai
682db25d265177367e35d5b2
Mingxing Li
682db25d265177367e35d5b3
Yong Liu
682db25d265177367e35d5b4
Jing Tang
682db25d265177367e35d5b5
Haoji Zhang
682db25d265177367e35d5b6
Lei Sun
682db25d265177367e35d5b7
Xiangxiang Chu
claimed_verified
2025-05-21T12:27:07.959Z
66d255e3947594430c723ff6
/avatars/c56e4792332a01bf34085a75ee64916e.svg
xiaochonglinghu
user
xiaochonglinghu
682db25d265177367e35d5b8
Yansong Tang
UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement
  Learning
Traditional visual grounding methods primarily focus on single-image
scenarios with simple textual references. However, extending these methods to
real-world scenarios that involve implicit and complex instructions,
particularly in conjunction with multiple images, poses significant challenges,
which is mainly due to the lack of advanced reasoning ability across diverse
multi-modal contexts. In this work, we aim to address the more practical
universal grounding task, and propose UniVG-R1, a reasoning guided multimodal
large language model (MLLM) for universal visual grounding, which enhances
reasoning capabilities through reinforcement learning (RL) combined with
cold-start data. Specifically, we first construct a high-quality
Chain-of-Thought (CoT) grounding dataset, annotated with detailed reasoning
chains, to guide the model towards correct reasoning paths via supervised
fine-tuning. Subsequently, we perform rule-based reinforcement learning to
encourage the model to identify correct reasoning chains, thereby incentivizing
its reasoning capabilities. In addition, we identify a difficulty bias arising
from the prevalence of easy samples as RL training progresses, and we propose a
difficulty-aware weight adjustment strategy to further strengthen the
performance. Experimental results demonstrate the effectiveness of UniVG-R1,
which achieves state-of-the-art performance on MIG-Bench with a 9.1%
improvement over the previous method. Furthermore, our model exhibits strong
generalizability, achieving an average improvement of 23.4% in zero-shot
performance across four image and video reasoning grounding benchmarks. The
project page can be accessed at https://amap-ml.github.io/UniVG-R1-page/.
UniVG-R1, a reasoning-guided multimodal large language model, enhances visual grounding by leveraging reinforcement learning and a difficulty-aware strategy, achieving state-of-the-art results and strong generalizability.
multimodal large language model reasoning guided reinforcement learning cold-start data Chain-of-Thought dataset supervised fine-tuning rule-based reinforcement learning difficulty bias difficulty-aware weight adjustment