SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware
  Reinforcement Learning
Multimodal large language models (MLLMs) have shown promising capabilities in
reasoning tasks, yet still struggle with complex problems requiring explicit
self-reflection and self-correction, especially compared to their unimodal
text-based counterparts. Existing reflection methods are simplistic and
struggle to generate meaningful and instructive feedback, as the reasoning
ability and knowledge limits of pre-trained models are largely fixed during
initial training. To overcome these challenges, we propose Multimodal
Self-Reflection enhanced reasoning with Group Relative Policy Optimization
(SRPO), a two-stage reflection-aware reinforcement learning (RL) framework
explicitly designed to enhance multimodal LLM reasoning. In the first stage, we
construct a high-quality, reflection-focused dataset under the guidance of an
advanced MLLM, which generates reflections based on initial responses to help
the policy model learn both reasoning and self-reflection. In the second stage,
we introduce a novel reward mechanism within the GRPO framework that encourages
concise and cognitively meaningful reflection while avoiding redundancy.
Extensive experiments across multiple multimodal reasoning benchmarks,
including MathVista, MathVision, MathVerse, and MMMU-Pro, using Qwen-2.5-VL-7B
and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms
state-of-the-art models, achieving notable improvements in both reasoning
accuracy and reflection quality.
683ec81753981b08324ce57b
Zhongwei Wan
683ec81753981b08324ce57c
Zhihao Dou
683ec81753981b08324ce57d
Che Liu
admin_assigned
2025-06-03T12:38:05.701Z
631b9ff5824f2502e3557c7e
/avatars/076043c9dba07644a570692563ef8114.svg
liu
user
che111
683ec81753981b08324ce57e
Yu Zhang
683ec81753981b08324ce57f
Dongfei Cui
admin_assigned
2025-06-03T12:38:16.192Z
64bd14f65b8d826146cd8512
/avatars/2b157e555495bac18ff332a309d0ce23.svg
Dongfei Cui
user
vencen
683ec81753981b08324ce580
Qinjian Zhao
claimed_verified
2025-06-20T12:20:21.482Z
6680dcacf8fd8217cc45f247
/avatars/cb1207f94587aa3e243492ddcc123264.svg
Albert Zhao 
user
Albert-CAC
683ec81753981b08324ce581
Hui Shen
683ec81753981b08324ce582
Jing Xiong
683ec81753981b08324ce583
Yi Xin
683ec81753981b08324ce584
Yifan Jiang
claimed_verified
2025-06-04T09:02:55.155Z
644078ea518271b0d1bf7ec6
/avatars/61014a3fd45f74ce541bdbe53929e233.svg
Yifan Jiang
user
YifanJ
683ec81753981b08324ce585
Yangfan He
683ec81753981b08324ce586
Mi Zhang
683ec81753981b08324ce587
Shen Yan
SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware
  Reinforcement Learning
Multimodal large language models (MLLMs) have shown promising capabilities in
reasoning tasks, yet still struggle with complex problems requiring explicit
self-reflection and self-correction, especially compared to their unimodal
text-based counterparts. Existing reflection methods are simplistic and
struggle to generate meaningful and instructive feedback, as the reasoning
ability and knowledge limits of pre-trained models are largely fixed during
initial training. To overcome these challenges, we propose Multimodal
Self-Reflection enhanced reasoning with Group Relative Policy Optimization
(SRPO), a two-stage reflection-aware reinforcement learning (RL) framework
explicitly designed to enhance multimodal LLM reasoning. In the first stage, we
construct a high-quality, reflection-focused dataset under the guidance of an
advanced MLLM, which generates reflections based on initial responses to help
the policy model learn both reasoning and self-reflection. In the second stage,
we introduce a novel reward mechanism within the GRPO framework that encourages
concise and cognitively meaningful reflection while avoiding redundancy.
Extensive experiments across multiple multimodal reasoning benchmarks,
including MathVista, MathVision, MathVerse, and MMMU-Pro, using Qwen-2.5-VL-7B
and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms
state-of-the-art models, achieving notable improvements in both reasoning
accuracy and reflection quality.
A two-stage reinforcement learning framework enhances reasoning and self-reflection in multimodal large language models through reflection-focused datasets and a novel reward mechanism.
multimodal large language models reasoning tasks reflection methods reinforcement learning reflection-aware reinforcement learning GRPO framework reward mechanism MathVista MathVision MathVerse MMMU-Pro Qwen-2.5-VL-7B Qwen-2.5-VL-32B reasoning accuracy reflection quality