Advancing Multimodal Reasoning via Reinforcement Learning with Cold
  Start
Recent advancements in large language models (LLMs) have demonstrated
impressive chain-of-thought reasoning capabilities, with reinforcement learning
(RL) playing a crucial role in this progress. While "aha moment"
patterns--where models exhibit self-correction through reflection--are often
attributed to emergent properties from RL, we first demonstrate that these
patterns exist in multimodal LLMs (MLLMs) prior to RL training but may not
necessarily correlate with improved reasoning performance. Building on these
insights, we present a comprehensive study on enhancing multimodal reasoning
through a two-stage approach: (1) supervised fine-tuning (SFT) as a cold start
with structured chain-of-thought reasoning patterns, followed by (2)
reinforcement learning via GRPO to further refine these capabilities. Our
extensive experiments show that this combined approach consistently outperforms
both SFT-only and RL-only methods across challenging multimodal reasoning
benchmarks. The resulting models achieve state-of-the-art performance among
open-source MLLMs at both 3B and 7B scales, with our 7B model showing
substantial improvements over base models (e.g., 66.3 %rightarrow73.4 % on
MathVista, 62.9 %rightarrow70.4 % on We-Math) and our 3B model achieving
performance competitive with several 7B models. Overall, this work provides
practical guidance for building advanced multimodal reasoning models. Our code
is available at https://github.com/waltonfuture/RL-with-Cold-Start.
6837c360b127cae8a0b36e85
Lai Wei
claimed_verified
2025-05-29T07:42:26.705Z
64a16b1aeacb4b50ba1c889d
https://cdn-avatars.huggingface.co/v1/production/uploads/64a16b1aeacb4b50ba1c889d/EhWoBGL6LlFGIVTUsp2F4.jpeg
Lai Wei
user
WaltonFuture
6837c360b127cae8a0b36e86
Yuting Li
claimed_verified
2025-06-10T11:00:30.177Z
66e3e801ec0343c253da37c3
/avatars/72fc67fa2b25473fd3acba1fbd132f62.svg
Yuting Li
user
Yuting6
6837c360b127cae8a0b36e87
Kaipeng Zheng
6837c360b127cae8a0b36e88
Chen Wang
6837c360b127cae8a0b36e89
Yue Wang
extracted_confirmed
2025-06-01T07:53:00.627Z
65e095da35ad8b2fe8c80e71
/avatars/225d4be6411dcec2460047ea1a88a5c3.svg
Weiran Huang
user
weiranhuang
6837c360b127cae8a0b36e8a
Linghe Kong
6837c360b127cae8a0b36e8b
Lichao Sun
6837c360b127cae8a0b36e8c
Weiran Huang
extracted_pending
2025-05-29T02:16:03.908Z
65e095da35ad8b2fe8c80e71
/avatars/225d4be6411dcec2460047ea1a88a5c3.svg
Weiran Huang
user
weiranhuang
Advancing Multimodal Reasoning via Reinforcement Learning with Cold
  Start
Recent advancements in large language models (LLMs) have demonstrated
impressive chain-of-thought reasoning capabilities, with reinforcement learning
(RL) playing a crucial role in this progress. While "aha moment"
patterns--where models exhibit self-correction through reflection--are often
attributed to emergent properties from RL, we first demonstrate that these
patterns exist in multimodal LLMs (MLLMs) prior to RL training but may not
necessarily correlate with improved reasoning performance. Building on these
insights, we present a comprehensive study on enhancing multimodal reasoning
through a two-stage approach: (1) supervised fine-tuning (SFT) as a cold start
with structured chain-of-thought reasoning patterns, followed by (2)
reinforcement learning via GRPO to further refine these capabilities. Our
extensive experiments show that this combined approach consistently outperforms
both SFT-only and RL-only methods across challenging multimodal reasoning
benchmarks. The resulting models achieve state-of-the-art performance among
open-source MLLMs at both 3B and 7B scales, with our 7B model showing
substantial improvements over base models (e.g., 66.3 %rightarrow73.4 % on
MathVista, 62.9 %rightarrow70.4 % on We-Math) and our 3B model achieving
performance competitive with several 7B models. Overall, this work provides
practical guidance for building advanced multimodal reasoning models. Our code
is available at https://github.com/waltonfuture/RL-with-Cold-Start.
A two-stage approach combining supervised fine-tuning and reinforcement learning enhances multimodal reasoning in large language models, achieving state-of-the-art performance on benchmarks.
large language models LLMs multimodal LLMs MLLMs reinforcement learning RL chain-of-thought reasoning GRPO supervised fine-tuning SFT multimodal reasoning MathVista We-Math