R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model
Recently DeepSeek R1 demonstrated how reinforcement learning with simple
rule-based incentives can enable autonomous development of complex reasoning in
large language models, characterized by the "aha moment", in which the model
manifest self-reflection and increased response length during training.
However, attempts to extend this success to multimodal reasoning often failed
to reproduce these key characteristics. In this report, we present the first
successful replication of these emergent characteristics for multimodal
reasoning on only a non-SFT 2B model. Starting with Qwen2-VL-2B and applying
reinforcement learning directly on the SAT dataset, our model achieves 59.47%
accuracy on CVBench, outperforming the base model by approximately ~30% and
exceeding both SFT setting by ~2%. In addition, we share our failed attempts
and insights in attempting to achieve R1-like reasoning using RL with instruct
models. aiming to shed light on the challenges involved. Our key observations
include: (1) applying RL on instruct model often results in trivial reasoning
trajectories, and (2) naive length reward are ineffective in eliciting
reasoning capabilities. The project code is available at
https://github.com/turningpoint-ai/VisualThinker-R1-Zero
67ce5ec17c6e6ea1cc5649c2
Hengguang Zhou
admin_assigned
2025-03-10T10:25:19.828Z
6565ebeda0623adbd76642f3
/avatars/5b11f4aabd82ce543ad8db0fe016a0f9.svg
Hengguang Zhou
user
Dolphin42
67ce5ec17c6e6ea1cc5649c3
Xirui Li
admin_assigned
2025-03-10T10:25:01.867Z
6534a434e778506c5b1e5be8
https://cdn-avatars.huggingface.co/v1/production/uploads/6534a434e778506c5b1e5be8/n_s7KqjSBOD8E31c1gEtw.jpeg
Xirui Li
user
AIcell
67ce5ec17c6e6ea1cc5649c4
Ruochen Wang
claimed_verified
2025-03-11T08:23:14.306Z
6376e832fe88a92b3a8cacfa
/avatars/1ae96b5c5d8b78a08173172dd5569d07.svg
Ruochen Wang
user
ruocwang
67ce5ec17c6e6ea1cc5649c5
Minhao Cheng
claimed_verified
2025-03-12T08:43:35.575Z
67cf1d6310dd1d670adf74ce
/avatars/15b3bef32acdc37093e2fc0b32a7b827.svg
Minhao Cheng
user
cmhcbb
67ce5ec17c6e6ea1cc5649c6
Tianyi Zhou
admin_assigned
2025-03-10T10:26:09.708Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
67ce5ec17c6e6ea1cc5649c7
Cho-Jui Hsieh
R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model
Recently DeepSeek R1 demonstrated how reinforcement learning with simple
rule-based incentives can enable autonomous development of complex reasoning in
large language models, characterized by the "aha moment", in which the model
manifest self-reflection and increased response length during training.
However, attempts to extend this success to multimodal reasoning often failed
to reproduce these key characteristics. In this report, we present the first
successful replication of these emergent characteristics for multimodal
reasoning on only a non-SFT 2B model. Starting with Qwen2-VL-2B and applying
reinforcement learning directly on the SAT dataset, our model achieves 59.47%
accuracy on CVBench, outperforming the base model by approximately ~30% and
exceeding both SFT setting by ~2%. In addition, we share our failed attempts
and insights in attempting to achieve R1-like reasoning using RL with instruct
models. aiming to shed light on the challenges involved. Our key observations
include: (1) applying RL on instruct model often results in trivial reasoning
trajectories, and (2) naive length reward are ineffective in eliciting
reasoning capabilities. The project code is available at
https://github.com/turningpoint-ai/VisualThinker-R1-Zero
A non-SFT model replicated emergent reasoning characteristics for multimodal tasks using reinforcement learning, achieving higher accuracy than base and SFT models on CVBench.
reinforcement learning large language models multimodal reasoning Qwen2-VL-2B SAT dataset CVBench instruct models trivial reasoning trajectories naive length reward