title: OThink-MR1: Stimulating multimodal generalized reasoning capabilities
    via dynamic reinforcement learning
summary: Multimodal Large Language Models (MLLMs) have gained significant traction for
  their ability to process diverse input data types and generate coherent,
  contextually relevant outputs across various applications. While supervised
  fine-tuning (SFT) has been the predominant approach to enhance MLLM
  capabilities in task-specific optimization, it often falls short in fostering
  crucial generalized reasoning abilities. Although reinforcement learning (RL)
  holds great promise in overcoming these limitations, it encounters two
  significant challenges: (1) its generalized capacities in multimodal tasks
  remain largely unexplored, and (2) its training constraints, including the
  constant Kullback-Leibler divergence or the clamp strategy, often result in
  suboptimal bottlenecks. To address these challenges, we propose OThink-MR1, an
  advanced MLLM equipped with profound comprehension and reasoning capabilities
  across multimodal tasks. Specifically, we introduce Group Relative Policy
  Optimization with a dynamic Kullback-Leibler strategy (GRPO-D), which markedly
  enhances reinforcement learning (RL) performance. For Qwen2-VL-2B-Instruct,
  GRPO-D achieves a relative improvement of more than 5.72% over SFT and more
  than 13.59% over GRPO in same-task evaluation on two adapted datasets.
  Furthermore, GRPO-D demonstrates remarkable cross-task generalization
  capabilities, with an average relative improvement of more than 61.63% over SFT
  in cross-task evaluation. These results highlight that the MLLM trained with
  GRPO-D on one multimodal task can be effectively transferred to another task,
  underscoring the superior generalized reasoning capabilities of our proposed
  OThink-MR1 model.
paper_authors._id: 67ea70d128179c61bef64043
paper_authors.name: Zhiyuan Liu
paper_authors._id: 67ea70d128179c61bef64044
paper_authors.name: Yuting Zhang
paper_authors._id: 67ea70d128179c61bef64045
paper_authors.name: Feng Liu
paper_authors._id: 67ea70d128179c61bef64046
paper_authors.name: Changwang Zhang
paper_authors._id: 67ea70d128179c61bef64047
paper_authors.name: Ying Sun
paper_authors._id: 67ea70d128179c61bef64048
paper_authors.name: Jun Wang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-31T10:39:14.597Z
paper_authors._id: 66da6c9e84f243eba3b49cf1
paper_authors.avatarUrl: /avatars/d30d96025141d68f18a28005a5d6f5af.svg
paper_authors.fullname: junwang.lu
paper_authors.type: user
paper_authors.user: jwanglux
paper_title: OThink-MR1: Stimulating multimodal generalized reasoning capabilities
    via dynamic reinforcement learning
paper_summary: Multimodal Large Language Models (MLLMs) have gained significant traction for
  their ability to process diverse input data types and generate coherent,
  contextually relevant outputs across various applications. While supervised
  fine-tuning (SFT) has been the predominant approach to enhance MLLM
  capabilities in task-specific optimization, it often falls short in fostering
  crucial generalized reasoning abilities. Although reinforcement learning (RL)
  holds great promise in overcoming these limitations, it encounters two
  significant challenges: (1) its generalized capacities in multimodal tasks
  remain largely unexplored, and (2) its training constraints, including the
  constant Kullback-Leibler divergence or the clamp strategy, often result in
  suboptimal bottlenecks. To address these challenges, we propose OThink-MR1, an
  advanced MLLM equipped with profound comprehension and reasoning capabilities
  across multimodal tasks. Specifically, we introduce Group Relative Policy
  Optimization with a dynamic Kullback-Leibler strategy (GRPO-D), which markedly
  enhances reinforcement learning (RL) performance. For Qwen2-VL-2B-Instruct,
  GRPO-D achieves a relative improvement of more than 5.72% over SFT and more
  than 13.59% over GRPO in same-task evaluation on two adapted datasets.
  Furthermore, GRPO-D demonstrates remarkable cross-task generalization
  capabilities, with an average relative improvement of more than 61.63% over SFT
  in cross-task evaluation. These results highlight that the MLLM trained with
  GRPO-D on one multimodal task can be effectively transferred to another task,
  underscoring the superior generalized reasoning capabilities of our proposed
  OThink-MR1 model.
paper_ai_summary: OThink-MR1, an advanced MLLM using GRPO-D, enhances reinforcement learning performance and demonstrates superior cross-task generalization compared to supervised fine-tuning.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) supervised fine-tuning (SFT) reinforcement learning (RL) Group Relative Policy Optimization (GRPO) dynamic Kullback-Leibler strategy (GRPO-D)