VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models
  with Reinforcement Learning
Recently, slow-thinking systems like GPT-o1 and DeepSeek-R1 have demonstrated
great potential in solving challenging problems through explicit reflection.
They significantly outperform the best fast-thinking models, such as GPT-4o, on
various math and science benchmarks. However, their multimodal reasoning
capabilities remain on par with fast-thinking models. For instance, GPT-o1's
performance on benchmarks like MathVista, MathVerse, and MathVision is similar
to fast-thinking models. In this paper, we aim to enhance the slow-thinking
capabilities of vision-language models using reinforcement learning (without
relying on distillation) to advance the state of the art. First, we adapt the
GRPO algorithm with a novel technique called Selective Sample Replay (SSR) to
address the vanishing advantages problem. While this approach yields strong
performance, the resulting RL-trained models exhibit limited self-reflection or
self-verification. To further encourage slow-thinking, we introduce Forced
Rethinking, which appends a textual rethinking trigger to the end of initial
rollouts in RL training, explicitly enforcing a self-reflection reasoning step.
By combining these two techniques, our model, VL-Rethinker, advances
state-of-the-art scores on MathVista, MathVerse, and MathVision to achieve
80.3%, 61.8%, and 43.9% respectively. VL-Rethinker also achieves open-source
SoTA on multi-disciplinary benchmarks such as MMMU-Pro, EMMA, and MEGA-Bench,
narrowing the gap with GPT-o1.
67fdc483ba0d61664fb0a19d
Haozhe Wang
claimed_verified
2025-04-15T07:54:20.455Z
65bf52f0259bc6caeb74f8bf
/avatars/b38392e954466df784a5760ded5df804.svg
Haozhe Wang
user
JasperHaozhe
67fdc483ba0d61664fb0a19e
Chao Qu
67fdc483ba0d61664fb0a19f
Zuming Huang
admin_assigned
2025-04-15T08:17:55.505Z
6772524ed6f92f429bd343a3
/avatars/211e0c4641b2d048b0136d7cdeef2483.svg
Zuming Huang
user
zuminghuang
67fdc483ba0d61664fb0a1a0
Wei Chu
67fdc483ba0d61664fb0a1a1
Fangzhen Lin
67fdc483ba0d61664fb0a1a2
Wenhu Chen
extracted_pending
2025-04-15T02:29:24.168Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models
  with Reinforcement Learning
Recently, slow-thinking systems like GPT-o1 and DeepSeek-R1 have demonstrated
great potential in solving challenging problems through explicit reflection.
They significantly outperform the best fast-thinking models, such as GPT-4o, on
various math and science benchmarks. However, their multimodal reasoning
capabilities remain on par with fast-thinking models. For instance, GPT-o1's
performance on benchmarks like MathVista, MathVerse, and MathVision is similar
to fast-thinking models. In this paper, we aim to enhance the slow-thinking
capabilities of vision-language models using reinforcement learning (without
relying on distillation) to advance the state of the art. First, we adapt the
GRPO algorithm with a novel technique called Selective Sample Replay (SSR) to
address the vanishing advantages problem. While this approach yields strong
performance, the resulting RL-trained models exhibit limited self-reflection or
self-verification. To further encourage slow-thinking, we introduce Forced
Rethinking, which appends a textual rethinking trigger to the end of initial
rollouts in RL training, explicitly enforcing a self-reflection reasoning step.
By combining these two techniques, our model, VL-Rethinker, advances
state-of-the-art scores on MathVista, MathVerse, and MathVision to achieve
80.3%, 61.8%, and 43.9% respectively. VL-Rethinker also achieves open-source
SoTA on multi-disciplinary benchmarks such as MMMU-Pro, EMMA, and MEGA-Bench,
narrowing the gap with GPT-o1.
Vision-language models enhanced with reinforcement learning and Forced Rethinking achieve state-of-the-art performance on math and science benchmarks and approach the capabilities of slow-thinking systems.
GRPO algorithm Selective Sample Replay Forced Rethinking VL-Rethinker MathVista MathVerse MathVision MMMU-Pro EMMA MEGA-Bench