Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models
  via Vision-Guided Reinforcement Learning
Large Vision-Language Models (LVLMs) typically follow a two-stage training
paradigm-pretraining and supervised fine-tuning. Recently, preference
optimization, derived from the language domain, has emerged as an effective
post-training reinforcement strategy to enhance capabilities of LVLMs. However,
constructing high-quality human-annotated preference data and developing robust
reward models to mimic these preferences are both costly and challenging.
Motivated by this observation, we propose Vision-R1, a novel vision-guided
R1-like reinforcement learning algorithm for LVLMs that rewards models with
definitive vision feedback. It only leverages curated instruction data,
eliminating the need for specialized reward models and handcrafted preference
datasets. We incorporate a criterion-driven reward function that further
integrates multi-dimensional feedback to evaluate model completions
comprehensively based on the vision task logic. Furthermore, we introduce a
progressive rule refinement strategy that dynamically adjusts the reward
criteria during training, enabling continuous model improvement and mitigating
reward hacking. Extensive experiments on both in-distribution and
out-of-distribution benchmarks demonstrate that fine-tuning the 7B LVLMs with
Vision-R1 achieves consistent performance gains, with even up to 50%
improvement and surpassing the state-of-the-art 10x size model.
67e22902af6628c90b525a2b
Yufei Zhan
67e22902af6628c90b525a2c
Yousong Zhu
67e22902af6628c90b525a2d
Shurong Zheng
67e22902af6628c90b525a2e
Hongyin Zhao
67e22902af6628c90b525a2f
Fan Yang
67e22902af6628c90b525a30
Ming Tang
67e22902af6628c90b525a31
Jinqiao Wang
Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models
  via Vision-Guided Reinforcement Learning
Large Vision-Language Models (LVLMs) typically follow a two-stage training
paradigm-pretraining and supervised fine-tuning. Recently, preference
optimization, derived from the language domain, has emerged as an effective
post-training reinforcement strategy to enhance capabilities of LVLMs. However,
constructing high-quality human-annotated preference data and developing robust
reward models to mimic these preferences are both costly and challenging.
Motivated by this observation, we propose Vision-R1, a novel vision-guided
R1-like reinforcement learning algorithm for LVLMs that rewards models with
definitive vision feedback. It only leverages curated instruction data,
eliminating the need for specialized reward models and handcrafted preference
datasets. We incorporate a criterion-driven reward function that further
integrates multi-dimensional feedback to evaluate model completions
comprehensively based on the vision task logic. Furthermore, we introduce a
progressive rule refinement strategy that dynamically adjusts the reward
criteria during training, enabling continuous model improvement and mitigating
reward hacking. Extensive experiments on both in-distribution and
out-of-distribution benchmarks demonstrate that fine-tuning the 7B LVLMs with
Vision-R1 achieves consistent performance gains, with even up to 50%
improvement and surpassing the state-of-the-art 10x size model.
Vision-R1 is a vision-guided reinforcement learning algorithm that enhances Large Vision-Language Models without requiring human-annotated preference data, achieving significant performance improvements.
Large Vision-Language Models LVLMs preference optimization reinforcement learning reward models human-annotated preference data vision-guided criterion-driven reward function multi-dimensional feedback progressive rule refinement reward hacking in-distribution benchmarks out-of-distribution benchmarks fine-tuning