title: Advancing Multimodal Reasoning via Reinforcement Learning with Cold
    Start
summary: Recent advancements in large language models (LLMs) have demonstrated
  impressive chain-of-thought reasoning capabilities, with reinforcement learning
  (RL) playing a crucial role in this progress. While "aha moment"
  patterns--where models exhibit self-correction through reflection--are often
  attributed to emergent properties from RL, we first demonstrate that these
  patterns exist in multimodal LLMs (MLLMs) prior to RL training but may not
  necessarily correlate with improved reasoning performance. Building on these
  insights, we present a comprehensive study on enhancing multimodal reasoning
  through a two-stage approach: (1) supervised fine-tuning (SFT) as a cold start
  with structured chain-of-thought reasoning patterns, followed by (2)
  reinforcement learning via GRPO to further refine these capabilities. Our
  extensive experiments show that this combined approach consistently outperforms
  both SFT-only and RL-only methods across challenging multimodal reasoning
  benchmarks. The resulting models achieve state-of-the-art performance among
  open-source MLLMs at both 3B and 7B scales, with our 7B model showing
  substantial improvements over base models (e.g., 66.3 %rightarrow73.4 % on
  MathVista, 62.9 %rightarrow70.4 % on We-Math) and our 3B model achieving
  performance competitive with several 7B models. Overall, this work provides
  practical guidance for building advanced multimodal reasoning models. Our code
  is available at https://github.com/waltonfuture/RL-with-Cold-Start.
paper_authors._id: 6837c360b127cae8a0b36e85
paper_authors.name: Lai Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:42:26.705Z
paper_authors._id: 64a16b1aeacb4b50ba1c889d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a16b1aeacb4b50ba1c889d/EhWoBGL6LlFGIVTUsp2F4.jpeg
paper_authors.fullname: Lai Wei
paper_authors.type: user
paper_authors.user: WaltonFuture
paper_authors._id: 6837c360b127cae8a0b36e86
paper_authors.name: Yuting Li
paper_authors._id: 6837c360b127cae8a0b36e87
paper_authors.name: Kaipeng Zheng
paper_authors._id: 6837c360b127cae8a0b36e88
paper_authors.name: Chen Wang
paper_authors._id: 6837c360b127cae8a0b36e89
paper_authors.name: Yue Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-06-01T07:53:00.627Z
paper_authors._id: 65e095da35ad8b2fe8c80e71
paper_authors.avatarUrl: /avatars/225d4be6411dcec2460047ea1a88a5c3.svg
paper_authors.fullname: Weiran Huang
paper_authors.type: user
paper_authors.user: weiranhuang
paper_authors._id: 6837c360b127cae8a0b36e8a
paper_authors.name: Linghe Kong
paper_authors._id: 6837c360b127cae8a0b36e8b
paper_authors.name: Lichao Sun
paper_authors._id: 6837c360b127cae8a0b36e8c
paper_authors.name: Weiran Huang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-29T02:16:03.908Z
paper_authors._id: 65e095da35ad8b2fe8c80e71
paper_authors.avatarUrl: /avatars/225d4be6411dcec2460047ea1a88a5c3.svg
paper_authors.fullname: Weiran Huang
paper_authors.type: user
paper_authors.user: weiranhuang
paper_title: Advancing Multimodal Reasoning via Reinforcement Learning with Cold
    Start
paper_summary: Recent advancements in large language models (LLMs) have demonstrated
  impressive chain-of-thought reasoning capabilities, with reinforcement learning
  (RL) playing a crucial role in this progress. While "aha moment"
  patterns--where models exhibit self-correction through reflection--are often
  attributed to emergent properties from RL, we first demonstrate that these
  patterns exist in multimodal LLMs (MLLMs) prior to RL training but may not
  necessarily correlate with improved reasoning performance. Building on these
  insights, we present a comprehensive study on enhancing multimodal reasoning
  through a two-stage approach: (1) supervised fine-tuning (SFT) as a cold start
  with structured chain-of-thought reasoning patterns, followed by (2)
  reinforcement learning via GRPO to further refine these capabilities. Our
  extensive experiments show that this combined approach consistently outperforms
  both SFT-only and RL-only methods across challenging multimodal reasoning
  benchmarks. The resulting models achieve state-of-the-art performance among
  open-source MLLMs at both 3B and 7B scales, with our 7B model showing
  substantial improvements over base models (e.g., 66.3 %rightarrow73.4 % on
  MathVista, 62.9 %rightarrow70.4 % on We-Math) and our 3B model achieving
  performance competitive with several 7B models. Overall, this work provides
  practical guidance for building advanced multimodal reasoning models. Our code
  is available at https://github.com/waltonfuture/RL-with-Cold-Start.
paper_ai_summary: A two-stage approach combining supervised fine-tuning and reinforcement learning enhances multimodal reasoning in large language models, achieving state-of-the-art performance on benchmarks.
paper_ai_keywords: large language models LLMs multimodal LLMs MLLMs reinforcement learning RL chain-of-thought reasoning GRPO supervised fine-tuning SFT multimodal reasoning MathVista We-Math