title: VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
summary: Recently DeepSeek R1 has shown that reinforcement learning (RL) can
  substantially improve the reasoning capabilities of Large Language Models
  (LLMs) through a simple yet effective design. The core of R1 lies in its
  rule-based reward formulation, which leverages tasks with deterministic
  ground-truth answers to enable precise and stable reward computation. In the
  visual domain, we similarly observe that a wide range of visual understanding
  tasks are inherently equipped with well-defined ground-truth annotations. This
  property makes them naturally compatible with rule-based reward mechanisms.
  Motivated by this observation, we investigate the extension of R1-style
  reinforcement learning to Vision-Language Models (VLMs), aiming to enhance
  their visual reasoning capabilities. To this end, we develop VLM-R1, a
  dedicated framework designed to harness RL for improving VLMs' performance on
  general vision-language tasks. Using this framework, we further explore the
  feasibility of applying RL to visual domain. Experimental results indicate that
  the RL-based model not only delivers competitive performance on visual
  understanding tasks but also surpasses Supervised Fine-Tuning (SFT) in
  generalization ability. Furthermore, we conduct comprehensive ablation studies
  that uncover a series of noteworthy insights, including the presence of reward
  hacking in object detection, the emergence of the "OD aha moment", the impact
  of training data quality, and the scaling behavior of RL across different model
  sizes. Through these analyses, we aim to deepen the understanding of how
  reinforcement learning enhances the capabilities of vision-language models, and
  we hope our findings and open-source contributions will support continued
  progress in the vision-language RL community. Our code and model are available
  at https://github.com/om-ai-lab/VLM-R1
paper_authors._id: 67fc52b190075b82590d75db
paper_authors.name: Haozhan Shen
paper_authors._id: 67fc52b190075b82590d75dc
paper_authors.name: Peng Liu
paper_authors._id: 67fc52b190075b82590d75dd
paper_authors.name: Jingcheng Li
paper_authors._id: 67fc52b190075b82590d75de
paper_authors.name: Chunxin Fang
paper_authors._id: 67fc52b190075b82590d75df
paper_authors.name: Yibo Ma
paper_authors._id: 67fc52b190075b82590d75e0
paper_authors.name: Jiajia Liao
paper_authors._id: 67fc52b190075b82590d75e1
paper_authors.name: Qiaoli Shen
paper_authors._id: 67fc52b190075b82590d75e2
paper_authors.name: Zilun Zhang
paper_authors._id: 67fc52b190075b82590d75e3
paper_authors.name: Kangjia Zhao
paper_authors._id: 67fc52b190075b82590d75e4
paper_authors.name: Qianqian Zhang
paper_authors._id: 67fc52b190075b82590d75e5
paper_authors.name: Ruochen Xu
paper_authors._id: 67fc52b190075b82590d75e6
paper_authors.name: Tiancheng Zhao
paper_title: VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
paper_summary: Recently DeepSeek R1 has shown that reinforcement learning (RL) can
  substantially improve the reasoning capabilities of Large Language Models
  (LLMs) through a simple yet effective design. The core of R1 lies in its
  rule-based reward formulation, which leverages tasks with deterministic
  ground-truth answers to enable precise and stable reward computation. In the
  visual domain, we similarly observe that a wide range of visual understanding
  tasks are inherently equipped with well-defined ground-truth annotations. This
  property makes them naturally compatible with rule-based reward mechanisms.
  Motivated by this observation, we investigate the extension of R1-style
  reinforcement learning to Vision-Language Models (VLMs), aiming to enhance
  their visual reasoning capabilities. To this end, we develop VLM-R1, a
  dedicated framework designed to harness RL for improving VLMs' performance on
  general vision-language tasks. Using this framework, we further explore the
  feasibility of applying RL to visual domain. Experimental results indicate that
  the RL-based model not only delivers competitive performance on visual
  understanding tasks but also surpasses Supervised Fine-Tuning (SFT) in
  generalization ability. Furthermore, we conduct comprehensive ablation studies
  that uncover a series of noteworthy insights, including the presence of reward
  hacking in object detection, the emergence of the "OD aha moment", the impact
  of training data quality, and the scaling behavior of RL across different model
  sizes. Through these analyses, we aim to deepen the understanding of how
  reinforcement learning enhances the capabilities of vision-language models, and
  we hope our findings and open-source contributions will support continued
  progress in the vision-language RL community. Our code and model are available
  at https://github.com/om-ai-lab/VLM-R1
paper_ai_summary: A reinforcement learning framework extends the capabilities of Vision-Language Models (VLMs) in visual reasoning by leveraging rule-based reward formulations, achieving competitive performance and superior generalization compared to supervised fine-tuning.
paper_ai_keywords: reinforcement learning Large Language Models (LLMs) rule-based reward formulation deterministic ground-truth answers Vision-Language Models (VLMs) VLM-R1 visual understanding tasks reward hacking OD aha moment training data quality scaling behavior