title: SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware
    Reinforcement Learning
summary: Multimodal large language models (MLLMs) have shown promising capabilities in
  reasoning tasks, yet still struggle with complex problems requiring explicit
  self-reflection and self-correction, especially compared to their unimodal
  text-based counterparts. Existing reflection methods are simplistic and
  struggle to generate meaningful and instructive feedback, as the reasoning
  ability and knowledge limits of pre-trained models are largely fixed during
  initial training. To overcome these challenges, we propose Multimodal
  Self-Reflection enhanced reasoning with Group Relative Policy Optimization
  (SRPO), a two-stage reflection-aware reinforcement learning (RL) framework
  explicitly designed to enhance multimodal LLM reasoning. In the first stage, we
  construct a high-quality, reflection-focused dataset under the guidance of an
  advanced MLLM, which generates reflections based on initial responses to help
  the policy model learn both reasoning and self-reflection. In the second stage,
  we introduce a novel reward mechanism within the GRPO framework that encourages
  concise and cognitively meaningful reflection while avoiding redundancy.
  Extensive experiments across multiple multimodal reasoning benchmarks,
  including MathVista, MathVision, MathVerse, and MMMU-Pro, using Qwen-2.5-VL-7B
  and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms
  state-of-the-art models, achieving notable improvements in both reasoning
  accuracy and reflection quality.
paper_authors._id: 683ec81753981b08324ce57b
paper_authors.name: Zhongwei Wan
paper_authors._id: 683ec81753981b08324ce57c
paper_authors.name: Zhihao Dou
paper_authors._id: 683ec81753981b08324ce57d
paper_authors.name: Che Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:38:05.701Z
paper_authors._id: 631b9ff5824f2502e3557c7e
paper_authors.avatarUrl: /avatars/076043c9dba07644a570692563ef8114.svg
paper_authors.fullname: liu
paper_authors.type: user
paper_authors.user: che111
paper_authors._id: 683ec81753981b08324ce57e
paper_authors.name: Yu Zhang
paper_authors._id: 683ec81753981b08324ce57f
paper_authors.name: Dongfei Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:38:16.192Z
paper_authors._id: 64bd14f65b8d826146cd8512
paper_authors.avatarUrl: /avatars/2b157e555495bac18ff332a309d0ce23.svg
paper_authors.fullname: Dongfei Cui
paper_authors.type: user
paper_authors.user: vencen
paper_authors._id: 683ec81753981b08324ce580
paper_authors.name: Qinjian Zhao
paper_authors._id: 683ec81753981b08324ce581
paper_authors.name: Hui Shen
paper_authors._id: 683ec81753981b08324ce582
paper_authors.name: Jing Xiong
paper_authors._id: 683ec81753981b08324ce583
paper_authors.name: Yi Xin
paper_authors._id: 683ec81753981b08324ce584
paper_authors.name: Yifan Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:02:55.155Z
paper_authors._id: 644078ea518271b0d1bf7ec6
paper_authors.avatarUrl: /avatars/61014a3fd45f74ce541bdbe53929e233.svg
paper_authors.fullname: Yifan Jiang
paper_authors.type: user
paper_authors.user: YifanJ
paper_authors._id: 683ec81753981b08324ce585
paper_authors.name: Yangfan He
paper_authors._id: 683ec81753981b08324ce586
paper_authors.name: Mi Zhang
paper_authors._id: 683ec81753981b08324ce587
paper_authors.name: Shen Yan
paper_title: SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware
    Reinforcement Learning
paper_summary: Multimodal large language models (MLLMs) have shown promising capabilities in
  reasoning tasks, yet still struggle with complex problems requiring explicit
  self-reflection and self-correction, especially compared to their unimodal
  text-based counterparts. Existing reflection methods are simplistic and
  struggle to generate meaningful and instructive feedback, as the reasoning
  ability and knowledge limits of pre-trained models are largely fixed during
  initial training. To overcome these challenges, we propose Multimodal
  Self-Reflection enhanced reasoning with Group Relative Policy Optimization
  (SRPO), a two-stage reflection-aware reinforcement learning (RL) framework
  explicitly designed to enhance multimodal LLM reasoning. In the first stage, we
  construct a high-quality, reflection-focused dataset under the guidance of an
  advanced MLLM, which generates reflections based on initial responses to help
  the policy model learn both reasoning and self-reflection. In the second stage,
  we introduce a novel reward mechanism within the GRPO framework that encourages
  concise and cognitively meaningful reflection while avoiding redundancy.
  Extensive experiments across multiple multimodal reasoning benchmarks,
  including MathVista, MathVision, MathVerse, and MMMU-Pro, using Qwen-2.5-VL-7B
  and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms
  state-of-the-art models, achieving notable improvements in both reasoning
  accuracy and reflection quality.
paper_ai_summary: A two-stage reinforcement learning framework enhances reasoning and self-reflection in multimodal large language models through reflection-focused datasets and a novel reward mechanism.
paper_ai_keywords: multimodal large language models reasoning tasks reflection methods reinforcement learning reflection-aware reinforcement learning GRPO framework reward mechanism MathVista MathVision MathVerse MMMU-Pro Qwen-2.5-VL-7B Qwen-2.5-VL-32B reasoning accuracy reflection quality