title: SFT or RL? An Early Investigation into Training R1-Like Reasoning Large
    Vision-Language Models
summary: This work revisits the dominant supervised fine-tuning (SFT) then
  reinforcement learning (RL) paradigm for training Large Vision-Language Models
  (LVLMs), and reveals a key finding: SFT can significantly undermine subsequent
  RL by inducing ``pseudo reasoning paths'' imitated from expert models. While
  these paths may resemble the native reasoning paths of RL models, they often
  involve prolonged, hesitant, less informative steps, and incorrect reasoning.
  To systematically study this effect, we introduce VLAA-Thinking, a new
  multimodal dataset designed to support reasoning in LVLMs. Constructed via a
  six-step pipeline involving captioning, reasoning distillation, answer rewrite
  and verification, VLAA-Thinking comprises high-quality, step-by-step visual
  reasoning traces for SFT, along with a more challenging RL split from the same
  data source. Using this dataset, we conduct extensive experiments comparing
  SFT, RL and their combinations. Results show that while SFT helps models learn
  reasoning formats, it often locks aligned models into imitative, rigid
  reasoning modes that impede further learning. In contrast, building on the
  Group Relative Policy Optimization (GRPO) with a novel mixed reward module
  integrating both perception and cognition signals, our RL approach fosters more
  genuine, adaptive reasoning behavior. Notably, our model VLAA-Thinker, based on
  Qwen2.5VL 3B, achieves top-1 performance on Open LMM Reasoning Leaderboard
  (https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard)
  among 4B scale LVLMs, surpassing the previous state-of-the-art by 1.8%. We hope
  our findings provide valuable insights in developing reasoning-capable LVLMs
  and can inform future research in this area.
paper_authors._id: 68008583bee5b086995074fe
paper_authors.name: Hardy Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-17T08:04:02.073Z
paper_authors._id: 65112b6b8c4b535a971ad3cf
paper_authors.avatarUrl: /avatars/1ef00b312d2b009c8c7aab21b4b3f258.svg
paper_authors.fullname: guiminghardychen
paper_authors.type: user
paper_authors.user: g-h-chen
paper_authors._id: 68008583bee5b086995074ff
paper_authors.name: Haoqin Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T07:57:03.666Z
paper_authors._id: 604ae011caabafacfa48e3de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
paper_authors.fullname: Haoqin Tu
paper_authors.type: user
paper_authors.user: PahaII
paper_authors._id: 68008583bee5b08699507500
paper_authors.name: Fali Wang
paper_authors._id: 68008583bee5b08699507501
paper_authors.name: Hui Liu
paper_authors._id: 68008583bee5b08699507502
paper_authors.name: Xianfeng Tang
paper_authors._id: 68008583bee5b08699507503
paper_authors.name: Xinya Du
paper_authors._id: 68008583bee5b08699507504
paper_authors.name: Yuyin Zhou
paper_authors._id: 68008583bee5b08699507505
paper_authors.name: Cihang Xie
paper_title: SFT or RL? An Early Investigation into Training R1-Like Reasoning Large
    Vision-Language Models
paper_summary: This work revisits the dominant supervised fine-tuning (SFT) then
  reinforcement learning (RL) paradigm for training Large Vision-Language Models
  (LVLMs), and reveals a key finding: SFT can significantly undermine subsequent
  RL by inducing ``pseudo reasoning paths'' imitated from expert models. While
  these paths may resemble the native reasoning paths of RL models, they often
  involve prolonged, hesitant, less informative steps, and incorrect reasoning.
  To systematically study this effect, we introduce VLAA-Thinking, a new
  multimodal dataset designed to support reasoning in LVLMs. Constructed via a
  six-step pipeline involving captioning, reasoning distillation, answer rewrite
  and verification, VLAA-Thinking comprises high-quality, step-by-step visual
  reasoning traces for SFT, along with a more challenging RL split from the same
  data source. Using this dataset, we conduct extensive experiments comparing
  SFT, RL and their combinations. Results show that while SFT helps models learn
  reasoning formats, it often locks aligned models into imitative, rigid
  reasoning modes that impede further learning. In contrast, building on the
  Group Relative Policy Optimization (GRPO) with a novel mixed reward module
  integrating both perception and cognition signals, our RL approach fosters more
  genuine, adaptive reasoning behavior. Notably, our model VLAA-Thinker, based on
  Qwen2.5VL 3B, achieves top-1 performance on Open LMM Reasoning Leaderboard
  (https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard)
  among 4B scale LVLMs, surpassing the previous state-of-the-art by 1.8%. We hope
  our findings provide valuable insights in developing reasoning-capable LVLMs
  and can inform future research in this area.
paper_ai_summary: Supervised fine-tuning can hinder reinforcement learning in Large Vision-Language Models by inducing imitative reasoning, whereas a novel reinforcement learning approach improves genuine reasoning capabilities and outperforms existing models.
paper_ai_keywords: supervised fine-tuning reinforcement learning Large Vision-Language Models pseudo reasoning paths VLAA-Thinking reasoning distillation answer rewrite verification Group Relative Policy Optimization mixed reward module perception signals cognition signals VLAA-Thinker Qwen2.5VL 3B Open LMM Reasoning Leaderboard