VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied
  Iterative Policy Optimization
Applying Reinforcement Learning (RL) to Video Large Language Models
(Video-LLMs) shows significant promise for complex video reasoning. However,
popular Reinforcement Fine-Tuning (RFT) methods, such as outcome-based Group
Relative Policy Optimization (GRPO), are limited by data preparation
bottlenecks (e.g., noise or high cost) and exhibit unstable improvements in the
quality of long chain-of-thoughts (CoTs) and downstream performance.To address
these limitations, we propose VerIPO, a Verifier-guided Iterative Policy
Optimization method designed to gradually improve video LLMs' capacity for
generating deep, long-term reasoning chains. The core component is
Rollout-Aware Verifier, positioned between the GRPO and Direct Preference
Optimization (DPO) training phases to form the GRPO-Verifier-DPO training loop.
This verifier leverages small LLMs as a judge to assess the reasoning logic of
rollouts, enabling the construction of high-quality contrastive data, including
reflective and contextually consistent CoTs. These curated preference samples
drive the efficient DPO stage (7x faster than GRPO), leading to marked
improvements in reasoning chain quality, especially in terms of length and
contextual consistency. This training loop benefits from GRPO's expansive
search and DPO's targeted optimization. Experimental results demonstrate: 1)
Significantly faster and more effective optimization compared to standard GRPO
variants, yielding superior performance; 2) Our trained models exceed the
direct inference of large-scale instruction-tuned Video-LLMs, producing long
and contextually consistent CoTs on diverse video reasoning tasks; and 3) Our
model with one iteration outperforms powerful LMMs (e.g., Kimi-VL) and long
reasoning models (e.g., Video-R1), highlighting its effectiveness and
stability.
683680e289cf929720599547
Yunxin Li
admin_assigned
2025-05-28T12:13:48.050Z
62fdb01bc1588e1d4c6c1a7c
/avatars/bd03085995b1c34e0ac8a845cf2c4e83.svg
Yunxin Li
user
YunxinLi
683680e289cf929720599548
Xinyu Chen
683680e289cf929720599549
Zitao Li
admin_assigned
2025-05-28T12:14:15.846Z
67ecc6a08647cfa1775a9fda
/avatars/bb15abd7a3d2c51380b0b1f819ef76e2.svg
Zitao Li
user
TerenceL-TL
683680e289cf92972059954a
Zhenyu Liu
683680e289cf92972059954b
Longyue Wang
admin_assigned
2025-05-28T12:14:32.185Z
636b030c328133bdb3a523bc
/avatars/15d5d5403fef2f1368bb4185b199061d.svg
Longyue Wang
user
longyuewang
683680e289cf92972059954c
Wenhan Luo
admin_assigned
2025-05-28T12:14:37.813Z
65f865cf806814aec87d45ce
/avatars/6b931c1efe20422a5985fce6a9e36aed.svg
Wenhan Luo
user
whluo
683680e289cf92972059954d
Baotian Hu
683680e289cf92972059954e
Min Zhang
VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied
  Iterative Policy Optimization
Applying Reinforcement Learning (RL) to Video Large Language Models
(Video-LLMs) shows significant promise for complex video reasoning. However,
popular Reinforcement Fine-Tuning (RFT) methods, such as outcome-based Group
Relative Policy Optimization (GRPO), are limited by data preparation
bottlenecks (e.g., noise or high cost) and exhibit unstable improvements in the
quality of long chain-of-thoughts (CoTs) and downstream performance.To address
these limitations, we propose VerIPO, a Verifier-guided Iterative Policy
Optimization method designed to gradually improve video LLMs' capacity for
generating deep, long-term reasoning chains. The core component is
Rollout-Aware Verifier, positioned between the GRPO and Direct Preference
Optimization (DPO) training phases to form the GRPO-Verifier-DPO training loop.
This verifier leverages small LLMs as a judge to assess the reasoning logic of
rollouts, enabling the construction of high-quality contrastive data, including
reflective and contextually consistent CoTs. These curated preference samples
drive the efficient DPO stage (7x faster than GRPO), leading to marked
improvements in reasoning chain quality, especially in terms of length and
contextual consistency. This training loop benefits from GRPO's expansive
search and DPO's targeted optimization. Experimental results demonstrate: 1)
Significantly faster and more effective optimization compared to standard GRPO
variants, yielding superior performance; 2) Our trained models exceed the
direct inference of large-scale instruction-tuned Video-LLMs, producing long
and contextually consistent CoTs on diverse video reasoning tasks; and 3) Our
model with one iteration outperforms powerful LMMs (e.g., Kimi-VL) and long
reasoning models (e.g., Video-R1), highlighting its effectiveness and
stability.
A Verifier-guided Iterative Policy Optimization method enhances Video-LLMs' reasoning capabilities by integrating a Rollout-Aware Verifier between GRPO and DPO phases, leading to faster and more effective optimization.
Reinforcement Learning Video Large Language Models Reinforcement Fine-Tuning Group Relative Policy Optimization Rollout-Aware Verifier Direct Preference Optimization long chain-of-thoughts video reasoning contrastive data reasoning chain quality contextual consistency