Semi-off-Policy Reinforcement Learning for Vision-Language Slow-thinking
  Reasoning
Enhancing large vision-language models (LVLMs) with visual slow-thinking
reasoning is crucial for solving complex multimodal tasks. However, since LVLMs
are mainly trained with vision-language alignment, it is difficult to adopt
on-policy reinforcement learning (RL) to develop the slow thinking ability
because the rollout space is restricted by its initial abilities. Off-policy RL
offers a way to go beyond the current policy, but directly distilling
trajectories from external models may cause visual hallucinations due to
mismatched visual perception abilities across models. To address these issues,
this paper proposes SOPHIA, a simple and scalable Semi-Off-Policy RL for
vision-language slow-tHInking reAsoning. SOPHIA builds a semi-off-policy
behavior model by combining on-policy visual understanding from a trainable
LVLM with off-policy slow-thinking reasoning from a language model, assigns
outcome-based rewards to reasoning, and propagates visual rewards backward.
Then LVLM learns slow-thinking reasoning ability from the obtained reasoning
trajectories using propagated rewards via off-policy RL algorithms. Extensive
experiments with InternVL2.5 and InternVL3.0 with 8B and 38B sizes show the
effectiveness of SOPHIA. Notably, SOPHIA improves InternVL3.0-38B by 8.50% in
average, reaching state-of-the-art performance among open-source LVLMs on
multiple multimodal reasoning benchmarks, and even outperforms some
closed-source models (e.g., GPT-4.1) on the challenging MathVision and
OlympiadBench, achieving 49.08% and 49.95% pass@1 accuracy, respectively.
Analysis shows SOPHIA outperforms supervised fine-tuning and direct on-policy
RL methods, offering a better policy initialization for further on-policy
training.
68805e7e00f4b5a05f2fbe6a
Junhao Shen
claimed_verified
2025-07-23T08:36:27.771Z
687f853bb39262ba84f3eeff
/avatars/5313f5974f289239488e842fe1536749.svg
Junhao Shen
user
shenjunhao
68805e7e00f4b5a05f2fbe6b
Haiteng Zhao
claimed_verified
2025-07-23T08:36:20.560Z
64a7c6e223622f7f189bcbe1
/avatars/4f13a7ed0d2b8d8dfff7dc650e46450a.svg
haiteng zhao
user
haitengzhao
68805e7e00f4b5a05f2fbe6c
Yuzhe Gu
claimed_verified
2025-07-23T08:36:25.759Z
6601196cc91ba4c08ad6e270
https://cdn-avatars.huggingface.co/v1/production/uploads/6601196cc91ba4c08ad6e270/X2YPNzUOQXBz5Gv-xR9LW.jpeg
yuzhe gu
user
vanilla1116
68805e7e00f4b5a05f2fbe6d
Songyang Gao
68805e7e00f4b5a05f2fbe6e
Kuikun Liu
claimed_verified
2025-07-23T08:36:22.475Z
63fd691794cc8f815d50c112
/avatars/87305d1cbfcc717e910ccdfaf0568f80.svg
liu
user
Harold-lkk
68805e7e00f4b5a05f2fbe6f
Haian Huang
68805e7e00f4b5a05f2fbe70
Jianfei Gao
68805e7e00f4b5a05f2fbe71
Dahua Lin
68805e7e00f4b5a05f2fbe72
Wenwei Zhang
68805e7e00f4b5a05f2fbe73
Kai Chen
Semi-off-Policy Reinforcement Learning for Vision-Language Slow-thinking
  Reasoning
Enhancing large vision-language models (LVLMs) with visual slow-thinking
reasoning is crucial for solving complex multimodal tasks. However, since LVLMs
are mainly trained with vision-language alignment, it is difficult to adopt
on-policy reinforcement learning (RL) to develop the slow thinking ability
because the rollout space is restricted by its initial abilities. Off-policy RL
offers a way to go beyond the current policy, but directly distilling
trajectories from external models may cause visual hallucinations due to
mismatched visual perception abilities across models. To address these issues,
this paper proposes SOPHIA, a simple and scalable Semi-Off-Policy RL for
vision-language slow-tHInking reAsoning. SOPHIA builds a semi-off-policy
behavior model by combining on-policy visual understanding from a trainable
LVLM with off-policy slow-thinking reasoning from a language model, assigns
outcome-based rewards to reasoning, and propagates visual rewards backward.
Then LVLM learns slow-thinking reasoning ability from the obtained reasoning
trajectories using propagated rewards via off-policy RL algorithms. Extensive
experiments with InternVL2.5 and InternVL3.0 with 8B and 38B sizes show the
effectiveness of SOPHIA. Notably, SOPHIA improves InternVL3.0-38B by 8.50% in
average, reaching state-of-the-art performance among open-source LVLMs on
multiple multimodal reasoning benchmarks, and even outperforms some
closed-source models (e.g., GPT-4.1) on the challenging MathVision and
OlympiadBench, achieving 49.08% and 49.95% pass@1 accuracy, respectively.
Analysis shows SOPHIA outperforms supervised fine-tuning and direct on-policy
RL methods, offering a better policy initialization for further on-policy
training.
SOPHIA, a semi-off-policy reinforcement learning approach, enhances large vision-language models with slow-thinking reasoning, improving performance on multimodal reasoning tasks.
large vision-language models visual slow-thinking reasoning on-policy reinforcement learning off-policy reinforcement learning visual hallucinations semi-off-policy behavior model outcome-based rewards visual rewards multimodal reasoning benchmarks MathVision OlympiadBench supervised fine-tuning