OThink-MR1: Stimulating multimodal generalized reasoning capabilities
  via dynamic reinforcement learning
Multimodal Large Language Models (MLLMs) have gained significant traction for
their ability to process diverse input data types and generate coherent,
contextually relevant outputs across various applications. While supervised
fine-tuning (SFT) has been the predominant approach to enhance MLLM
capabilities in task-specific optimization, it often falls short in fostering
crucial generalized reasoning abilities. Although reinforcement learning (RL)
holds great promise in overcoming these limitations, it encounters two
significant challenges: (1) its generalized capacities in multimodal tasks
remain largely unexplored, and (2) its training constraints, including the
constant Kullback-Leibler divergence or the clamp strategy, often result in
suboptimal bottlenecks. To address these challenges, we propose OThink-MR1, an
advanced MLLM equipped with profound comprehension and reasoning capabilities
across multimodal tasks. Specifically, we introduce Group Relative Policy
Optimization with a dynamic Kullback-Leibler strategy (GRPO-D), which markedly
enhances reinforcement learning (RL) performance. For Qwen2-VL-2B-Instruct,
GRPO-D achieves a relative improvement of more than 5.72% over SFT and more
than 13.59% over GRPO in same-task evaluation on two adapted datasets.
Furthermore, GRPO-D demonstrates remarkable cross-task generalization
capabilities, with an average relative improvement of more than 61.63% over SFT
in cross-task evaluation. These results highlight that the MLLM trained with
GRPO-D on one multimodal task can be effectively transferred to another task,
underscoring the superior generalized reasoning capabilities of our proposed
OThink-MR1 model.
67ea70d128179c61bef64043
Zhiyuan Liu
67ea70d128179c61bef64044
Yuting Zhang
67ea70d128179c61bef64045
Feng Liu
67ea70d128179c61bef64046
Changwang Zhang
67ea70d128179c61bef64047
Ying Sun
67ea70d128179c61bef64048
Jun Wang
extracted_pending
2025-03-31T10:39:14.597Z
66da6c9e84f243eba3b49cf1
/avatars/d30d96025141d68f18a28005a5d6f5af.svg
junwang.lu
user
jwanglux
OThink-MR1: Stimulating multimodal generalized reasoning capabilities
  via dynamic reinforcement learning
Multimodal Large Language Models (MLLMs) have gained significant traction for
their ability to process diverse input data types and generate coherent,
contextually relevant outputs across various applications. While supervised
fine-tuning (SFT) has been the predominant approach to enhance MLLM
capabilities in task-specific optimization, it often falls short in fostering
crucial generalized reasoning abilities. Although reinforcement learning (RL)
holds great promise in overcoming these limitations, it encounters two
significant challenges: (1) its generalized capacities in multimodal tasks
remain largely unexplored, and (2) its training constraints, including the
constant Kullback-Leibler divergence or the clamp strategy, often result in
suboptimal bottlenecks. To address these challenges, we propose OThink-MR1, an
advanced MLLM equipped with profound comprehension and reasoning capabilities
across multimodal tasks. Specifically, we introduce Group Relative Policy
Optimization with a dynamic Kullback-Leibler strategy (GRPO-D), which markedly
enhances reinforcement learning (RL) performance. For Qwen2-VL-2B-Instruct,
GRPO-D achieves a relative improvement of more than 5.72% over SFT and more
than 13.59% over GRPO in same-task evaluation on two adapted datasets.
Furthermore, GRPO-D demonstrates remarkable cross-task generalization
capabilities, with an average relative improvement of more than 61.63% over SFT
in cross-task evaluation. These results highlight that the MLLM trained with
GRPO-D on one multimodal task can be effectively transferred to another task,
underscoring the superior generalized reasoning capabilities of our proposed
OThink-MR1 model.
OThink-MR1, an advanced MLLM using GRPO-D, enhances reinforcement learning performance and demonstrates superior cross-task generalization compared to supervised fine-tuning.
Multimodal Large Language Models (MLLMs) supervised fine-tuning (SFT) reinforcement learning (RL) Group Relative Policy Optimization (GRPO) dynamic Kullback-Leibler strategy (GRPO-D)