title: R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model
summary: Recently DeepSeek R1 demonstrated how reinforcement learning with simple
  rule-based incentives can enable autonomous development of complex reasoning in
  large language models, characterized by the "aha moment", in which the model
  manifest self-reflection and increased response length during training.
  However, attempts to extend this success to multimodal reasoning often failed
  to reproduce these key characteristics. In this report, we present the first
  successful replication of these emergent characteristics for multimodal
  reasoning on only a non-SFT 2B model. Starting with Qwen2-VL-2B and applying
  reinforcement learning directly on the SAT dataset, our model achieves 59.47%
  accuracy on CVBench, outperforming the base model by approximately ~30% and
  exceeding both SFT setting by ~2%. In addition, we share our failed attempts
  and insights in attempting to achieve R1-like reasoning using RL with instruct
  models. aiming to shed light on the challenges involved. Our key observations
  include: (1) applying RL on instruct model often results in trivial reasoning
  trajectories, and (2) naive length reward are ineffective in eliciting
  reasoning capabilities. The project code is available at
  https://github.com/turningpoint-ai/VisualThinker-R1-Zero
paper_authors._id: 67ce5ec17c6e6ea1cc5649c2
paper_authors.name: Hengguang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T10:25:19.828Z
paper_authors._id: 6565ebeda0623adbd76642f3
paper_authors.avatarUrl: /avatars/5b11f4aabd82ce543ad8db0fe016a0f9.svg
paper_authors.fullname: Hengguang Zhou
paper_authors.type: user
paper_authors.user: Dolphin42
paper_authors._id: 67ce5ec17c6e6ea1cc5649c3
paper_authors.name: Xirui Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T10:25:01.867Z
paper_authors._id: 6534a434e778506c5b1e5be8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6534a434e778506c5b1e5be8/n_s7KqjSBOD8E31c1gEtw.jpeg
paper_authors.fullname: Xirui Li
paper_authors.type: user
paper_authors.user: AIcell
paper_authors._id: 67ce5ec17c6e6ea1cc5649c4
paper_authors.name: Ruochen Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:23:14.306Z
paper_authors._id: 6376e832fe88a92b3a8cacfa
paper_authors.avatarUrl: /avatars/1ae96b5c5d8b78a08173172dd5569d07.svg
paper_authors.fullname: Ruochen Wang
paper_authors.type: user
paper_authors.user: ruocwang
paper_authors._id: 67ce5ec17c6e6ea1cc5649c5
paper_authors.name: Minhao Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:43:35.575Z
paper_authors._id: 67cf1d6310dd1d670adf74ce
paper_authors.avatarUrl: /avatars/15b3bef32acdc37093e2fc0b32a7b827.svg
paper_authors.fullname: Minhao Cheng
paper_authors.type: user
paper_authors.user: cmhcbb
paper_authors._id: 67ce5ec17c6e6ea1cc5649c6
paper_authors.name: Tianyi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T10:26:09.708Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_authors._id: 67ce5ec17c6e6ea1cc5649c7
paper_authors.name: Cho-Jui Hsieh
paper_title: R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model
paper_summary: Recently DeepSeek R1 demonstrated how reinforcement learning with simple
  rule-based incentives can enable autonomous development of complex reasoning in
  large language models, characterized by the "aha moment", in which the model
  manifest self-reflection and increased response length during training.
  However, attempts to extend this success to multimodal reasoning often failed
  to reproduce these key characteristics. In this report, we present the first
  successful replication of these emergent characteristics for multimodal
  reasoning on only a non-SFT 2B model. Starting with Qwen2-VL-2B and applying
  reinforcement learning directly on the SAT dataset, our model achieves 59.47%
  accuracy on CVBench, outperforming the base model by approximately ~30% and
  exceeding both SFT setting by ~2%. In addition, we share our failed attempts
  and insights in attempting to achieve R1-like reasoning using RL with instruct
  models. aiming to shed light on the challenges involved. Our key observations
  include: (1) applying RL on instruct model often results in trivial reasoning
  trajectories, and (2) naive length reward are ineffective in eliciting
  reasoning capabilities. The project code is available at
  https://github.com/turningpoint-ai/VisualThinker-R1-Zero
paper_ai_summary: A non-SFT model replicated emergent reasoning characteristics for multimodal tasks using reinforcement learning, achieving higher accuracy than base and SFT models on CVBench.
paper_ai_keywords: reinforcement learning large language models multimodal reasoning Qwen2-VL-2B SAT dataset CVBench instruct models trivial reasoning trajectories naive length reward