title: R1-VL: Learning to Reason with Multimodal Large Language Models via
    Step-wise Group Relative Policy Optimization
summary: Recent studies generally enhance MLLMs' reasoning capabilities via supervised
  fine-tuning on high-quality chain-of-thought reasoning data, which often leads
  models to merely imitate successful reasoning paths without understanding what
  the wrong reasoning paths are. In this work, we aim to enhance the MLLMs'
  reasoning ability beyond passively imitating positive reasoning paths. To this
  end, we design Step-wise Group Relative Policy Optimization (StepGRPO), a new
  online reinforcement learning framework that enables MLLMs to self-improve
  reasoning ability via simple, effective and dense step-wise rewarding.
  Specifically, StepGRPO introduces two novel rule-based reasoning rewards:
  Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity
  Reward (StepRVR). StepRAR rewards the reasoning paths that contain necessary
  intermediate reasoning steps via a soft key-step matching technique, while
  StepRAR rewards reasoning paths that follow a well-structured and logically
  consistent reasoning process through a reasoning completeness and logic
  evaluation strategy. With the proposed StepGRPO, we introduce R1-VL, a series
  of MLLMs with outstanding capabilities in step-by-step reasoning. Extensive
  experiments over 8 benchmarks demonstrate the superiority of our methods.
paper_authors._id: 67d8eb0c18de6ef86c4eb457
paper_authors.name: Jingyi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T08:54:57.692Z
paper_authors._id: 67d7d1bc1414fcb61968daed
paper_authors.avatarUrl: /avatars/df3d10511a73ab6370d6df90f2759489.svg
paper_authors.fullname: Jingyi Zhang
paper_authors.type: user
paper_authors.user: jingyiZ00
paper_authors._id: 67d8eb0c18de6ef86c4eb458
paper_authors.name: Jiaxing Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:31:01.731Z
paper_authors._id: 65237910b80dc49ba03a96d9
paper_authors.avatarUrl: /avatars/9d81c4c8fb2d597079e8dd9d9b79a8d8.svg
paper_authors.fullname: jiaxing
paper_authors.type: user
paper_authors.user: huangjiaxing
paper_authors._id: 67d8eb0c18de6ef86c4eb459
paper_authors.name: Huanjin Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:31:08.069Z
paper_authors._id: 6590e03454f8826173ed5ee6
paper_authors.avatarUrl: /avatars/f5e59d3e58c28a99f2ff39267ca51cdb.svg
paper_authors.fullname: Huanjin Yao
paper_authors.type: user
paper_authors.user: HuanjinYao
paper_authors._id: 67d8eb0c18de6ef86c4eb45a
paper_authors.name: Shunyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:31:15.587Z
paper_authors._id: 6713afea187a20dc579e121b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6713afea187a20dc579e121b/ELxVQLVF9ifuT-TCWPK22.jpeg
paper_authors.fullname: Shunyu Liu
paper_authors.type: user
paper_authors.user: liushunyu
paper_authors._id: 67d8eb0c18de6ef86c4eb45b
paper_authors.name: Xikun Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:31:21.950Z
paper_authors._id: 6274a9620d11b4f675085fbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1651812606924-noauth.jpeg
paper_authors.fullname: Xikun Zhang
paper_authors.type: user
paper_authors.user: Xikun
paper_authors._id: 67d8eb0c18de6ef86c4eb45c
paper_authors.name: Shijian Lu
paper_authors._id: 67d8eb0c18de6ef86c4eb45d
paper_authors.name: Dacheng Tao
paper_title: R1-VL: Learning to Reason with Multimodal Large Language Models via
    Step-wise Group Relative Policy Optimization
paper_summary: Recent studies generally enhance MLLMs' reasoning capabilities via supervised
  fine-tuning on high-quality chain-of-thought reasoning data, which often leads
  models to merely imitate successful reasoning paths without understanding what
  the wrong reasoning paths are. In this work, we aim to enhance the MLLMs'
  reasoning ability beyond passively imitating positive reasoning paths. To this
  end, we design Step-wise Group Relative Policy Optimization (StepGRPO), a new
  online reinforcement learning framework that enables MLLMs to self-improve
  reasoning ability via simple, effective and dense step-wise rewarding.
  Specifically, StepGRPO introduces two novel rule-based reasoning rewards:
  Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity
  Reward (StepRVR). StepRAR rewards the reasoning paths that contain necessary
  intermediate reasoning steps via a soft key-step matching technique, while
  StepRAR rewards reasoning paths that follow a well-structured and logically
  consistent reasoning process through a reasoning completeness and logic
  evaluation strategy. With the proposed StepGRPO, we introduce R1-VL, a series
  of MLLMs with outstanding capabilities in step-by-step reasoning. Extensive
  experiments over 8 benchmarks demonstrate the superiority of our methods.
paper_ai_summary: A reinforcement learning framework enhances MLLMs' reasoning by rewarding step-wise accurate and valid reasoning paths, leading to improved performance on reasoning benchmarks.
paper_ai_keywords: MLLMs reasoning capabilities supervised fine-tuning reinforcement learning Step-wise Group Relative Policy Optimization (StepGRPO) Step-wise Reasoning Accuracy Reward (StepRAR) Step-wise Reasoning Validity Reward (StepRVR) soft key-step matching reasoning completeness logic evaluation