title: Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models
    via Vision-Guided Reinforcement Learning
summary: Large Vision-Language Models (LVLMs) typically follow a two-stage training
  paradigm-pretraining and supervised fine-tuning. Recently, preference
  optimization, derived from the language domain, has emerged as an effective
  post-training reinforcement strategy to enhance capabilities of LVLMs. However,
  constructing high-quality human-annotated preference data and developing robust
  reward models to mimic these preferences are both costly and challenging.
  Motivated by this observation, we propose Vision-R1, a novel vision-guided
  R1-like reinforcement learning algorithm for LVLMs that rewards models with
  definitive vision feedback. It only leverages curated instruction data,
  eliminating the need for specialized reward models and handcrafted preference
  datasets. We incorporate a criterion-driven reward function that further
  integrates multi-dimensional feedback to evaluate model completions
  comprehensively based on the vision task logic. Furthermore, we introduce a
  progressive rule refinement strategy that dynamically adjusts the reward
  criteria during training, enabling continuous model improvement and mitigating
  reward hacking. Extensive experiments on both in-distribution and
  out-of-distribution benchmarks demonstrate that fine-tuning the 7B LVLMs with
  Vision-R1 achieves consistent performance gains, with even up to 50%
  improvement and surpassing the state-of-the-art 10x size model.
paper_authors._id: 67e22902af6628c90b525a2b
paper_authors.name: Yufei Zhan
paper_authors._id: 67e22902af6628c90b525a2c
paper_authors.name: Yousong Zhu
paper_authors._id: 67e22902af6628c90b525a2d
paper_authors.name: Shurong Zheng
paper_authors._id: 67e22902af6628c90b525a2e
paper_authors.name: Hongyin Zhao
paper_authors._id: 67e22902af6628c90b525a2f
paper_authors.name: Fan Yang
paper_authors._id: 67e22902af6628c90b525a30
paper_authors.name: Ming Tang
paper_authors._id: 67e22902af6628c90b525a31
paper_authors.name: Jinqiao Wang
paper_title: Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models
    via Vision-Guided Reinforcement Learning
paper_summary: Large Vision-Language Models (LVLMs) typically follow a two-stage training
  paradigm-pretraining and supervised fine-tuning. Recently, preference
  optimization, derived from the language domain, has emerged as an effective
  post-training reinforcement strategy to enhance capabilities of LVLMs. However,
  constructing high-quality human-annotated preference data and developing robust
  reward models to mimic these preferences are both costly and challenging.
  Motivated by this observation, we propose Vision-R1, a novel vision-guided
  R1-like reinforcement learning algorithm for LVLMs that rewards models with
  definitive vision feedback. It only leverages curated instruction data,
  eliminating the need for specialized reward models and handcrafted preference
  datasets. We incorporate a criterion-driven reward function that further
  integrates multi-dimensional feedback to evaluate model completions
  comprehensively based on the vision task logic. Furthermore, we introduce a
  progressive rule refinement strategy that dynamically adjusts the reward
  criteria during training, enabling continuous model improvement and mitigating
  reward hacking. Extensive experiments on both in-distribution and
  out-of-distribution benchmarks demonstrate that fine-tuning the 7B LVLMs with
  Vision-R1 achieves consistent performance gains, with even up to 50%
  improvement and surpassing the state-of-the-art 10x size model.
paper_ai_summary: Vision-R1 is a vision-guided reinforcement learning algorithm that enhances Large Vision-Language Models without requiring human-annotated preference data, achieving significant performance improvements.
paper_ai_keywords: Large Vision-Language Models LVLMs preference optimization reinforcement learning reward models human-annotated preference data vision-guided criterion-driven reward function multi-dimensional feedback progressive rule refinement reward hacking in-distribution benchmarks out-of-distribution benchmarks fine-tuning