title: VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models
    with Reinforcement Learning
summary: Recently, slow-thinking systems like GPT-o1 and DeepSeek-R1 have demonstrated
  great potential in solving challenging problems through explicit reflection.
  They significantly outperform the best fast-thinking models, such as GPT-4o, on
  various math and science benchmarks. However, their multimodal reasoning
  capabilities remain on par with fast-thinking models. For instance, GPT-o1's
  performance on benchmarks like MathVista, MathVerse, and MathVision is similar
  to fast-thinking models. In this paper, we aim to enhance the slow-thinking
  capabilities of vision-language models using reinforcement learning (without
  relying on distillation) to advance the state of the art. First, we adapt the
  GRPO algorithm with a novel technique called Selective Sample Replay (SSR) to
  address the vanishing advantages problem. While this approach yields strong
  performance, the resulting RL-trained models exhibit limited self-reflection or
  self-verification. To further encourage slow-thinking, we introduce Forced
  Rethinking, which appends a textual rethinking trigger to the end of initial
  rollouts in RL training, explicitly enforcing a self-reflection reasoning step.
  By combining these two techniques, our model, VL-Rethinker, advances
  state-of-the-art scores on MathVista, MathVerse, and MathVision to achieve
  80.3%, 61.8%, and 43.9% respectively. VL-Rethinker also achieves open-source
  SoTA on multi-disciplinary benchmarks such as MMMU-Pro, EMMA, and MEGA-Bench,
  narrowing the gap with GPT-o1.
paper_authors._id: 67fdc483ba0d61664fb0a19d
paper_authors.name: Haozhe Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:54:20.455Z
paper_authors._id: 65bf52f0259bc6caeb74f8bf
paper_authors.avatarUrl: /avatars/b38392e954466df784a5760ded5df804.svg
paper_authors.fullname: Haozhe Wang
paper_authors.type: user
paper_authors.user: JasperHaozhe
paper_authors._id: 67fdc483ba0d61664fb0a19e
paper_authors.name: Chao Qu
paper_authors._id: 67fdc483ba0d61664fb0a19f
paper_authors.name: Zuming Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:17:55.505Z
paper_authors._id: 6772524ed6f92f429bd343a3
paper_authors.avatarUrl: /avatars/211e0c4641b2d048b0136d7cdeef2483.svg
paper_authors.fullname: Zuming Huang
paper_authors.type: user
paper_authors.user: zuminghuang
paper_authors._id: 67fdc483ba0d61664fb0a1a0
paper_authors.name: Wei Chu
paper_authors._id: 67fdc483ba0d61664fb0a1a1
paper_authors.name: Fangzhen Lin
paper_authors._id: 67fdc483ba0d61664fb0a1a2
paper_authors.name: Wenhu Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-15T02:29:24.168Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models
    with Reinforcement Learning
paper_summary: Recently, slow-thinking systems like GPT-o1 and DeepSeek-R1 have demonstrated
  great potential in solving challenging problems through explicit reflection.
  They significantly outperform the best fast-thinking models, such as GPT-4o, on
  various math and science benchmarks. However, their multimodal reasoning
  capabilities remain on par with fast-thinking models. For instance, GPT-o1's
  performance on benchmarks like MathVista, MathVerse, and MathVision is similar
  to fast-thinking models. In this paper, we aim to enhance the slow-thinking
  capabilities of vision-language models using reinforcement learning (without
  relying on distillation) to advance the state of the art. First, we adapt the
  GRPO algorithm with a novel technique called Selective Sample Replay (SSR) to
  address the vanishing advantages problem. While this approach yields strong
  performance, the resulting RL-trained models exhibit limited self-reflection or
  self-verification. To further encourage slow-thinking, we introduce Forced
  Rethinking, which appends a textual rethinking trigger to the end of initial
  rollouts in RL training, explicitly enforcing a self-reflection reasoning step.
  By combining these two techniques, our model, VL-Rethinker, advances
  state-of-the-art scores on MathVista, MathVerse, and MathVision to achieve
  80.3%, 61.8%, and 43.9% respectively. VL-Rethinker also achieves open-source
  SoTA on multi-disciplinary benchmarks such as MMMU-Pro, EMMA, and MEGA-Bench,
  narrowing the gap with GPT-o1.
paper_ai_summary: Vision-language models enhanced with reinforcement learning and Forced Rethinking achieve state-of-the-art performance on math and science benchmarks and approach the capabilities of slow-thinking systems.
paper_ai_keywords: GRPO algorithm Selective Sample Replay Forced Rethinking VL-Rethinker MathVista MathVerse MathVision MMMU-Pro EMMA MEGA-Bench