title: VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied
    Iterative Policy Optimization
summary: Applying Reinforcement Learning (RL) to Video Large Language Models
  (Video-LLMs) shows significant promise for complex video reasoning. However,
  popular Reinforcement Fine-Tuning (RFT) methods, such as outcome-based Group
  Relative Policy Optimization (GRPO), are limited by data preparation
  bottlenecks (e.g., noise or high cost) and exhibit unstable improvements in the
  quality of long chain-of-thoughts (CoTs) and downstream performance.To address
  these limitations, we propose VerIPO, a Verifier-guided Iterative Policy
  Optimization method designed to gradually improve video LLMs' capacity for
  generating deep, long-term reasoning chains. The core component is
  Rollout-Aware Verifier, positioned between the GRPO and Direct Preference
  Optimization (DPO) training phases to form the GRPO-Verifier-DPO training loop.
  This verifier leverages small LLMs as a judge to assess the reasoning logic of
  rollouts, enabling the construction of high-quality contrastive data, including
  reflective and contextually consistent CoTs. These curated preference samples
  drive the efficient DPO stage (7x faster than GRPO), leading to marked
  improvements in reasoning chain quality, especially in terms of length and
  contextual consistency. This training loop benefits from GRPO's expansive
  search and DPO's targeted optimization. Experimental results demonstrate: 1)
  Significantly faster and more effective optimization compared to standard GRPO
  variants, yielding superior performance; 2) Our trained models exceed the
  direct inference of large-scale instruction-tuned Video-LLMs, producing long
  and contextually consistent CoTs on diverse video reasoning tasks; and 3) Our
  model with one iteration outperforms powerful LMMs (e.g., Kimi-VL) and long
  reasoning models (e.g., Video-R1), highlighting its effectiveness and
  stability.
paper_authors._id: 683680e289cf929720599547
paper_authors.name: Yunxin Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:13:48.050Z
paper_authors._id: 62fdb01bc1588e1d4c6c1a7c
paper_authors.avatarUrl: /avatars/bd03085995b1c34e0ac8a845cf2c4e83.svg
paper_authors.fullname: Yunxin Li
paper_authors.type: user
paper_authors.user: YunxinLi
paper_authors._id: 683680e289cf929720599548
paper_authors.name: Xinyu Chen
paper_authors._id: 683680e289cf929720599549
paper_authors.name: Zitao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:14:15.846Z
paper_authors._id: 67ecc6a08647cfa1775a9fda
paper_authors.avatarUrl: /avatars/bb15abd7a3d2c51380b0b1f819ef76e2.svg
paper_authors.fullname: Zitao Li
paper_authors.type: user
paper_authors.user: TerenceL-TL
paper_authors._id: 683680e289cf92972059954a
paper_authors.name: Zhenyu Liu
paper_authors._id: 683680e289cf92972059954b
paper_authors.name: Longyue Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:14:32.185Z
paper_authors._id: 636b030c328133bdb3a523bc
paper_authors.avatarUrl: /avatars/15d5d5403fef2f1368bb4185b199061d.svg
paper_authors.fullname: Longyue Wang
paper_authors.type: user
paper_authors.user: longyuewang
paper_authors._id: 683680e289cf92972059954c
paper_authors.name: Wenhan Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:14:37.813Z
paper_authors._id: 65f865cf806814aec87d45ce
paper_authors.avatarUrl: /avatars/6b931c1efe20422a5985fce6a9e36aed.svg
paper_authors.fullname: Wenhan Luo
paper_authors.type: user
paper_authors.user: whluo
paper_authors._id: 683680e289cf92972059954d
paper_authors.name: Baotian Hu
paper_authors._id: 683680e289cf92972059954e
paper_authors.name: Min Zhang
paper_title: VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied
    Iterative Policy Optimization
paper_summary: Applying Reinforcement Learning (RL) to Video Large Language Models
  (Video-LLMs) shows significant promise for complex video reasoning. However,
  popular Reinforcement Fine-Tuning (RFT) methods, such as outcome-based Group
  Relative Policy Optimization (GRPO), are limited by data preparation
  bottlenecks (e.g., noise or high cost) and exhibit unstable improvements in the
  quality of long chain-of-thoughts (CoTs) and downstream performance.To address
  these limitations, we propose VerIPO, a Verifier-guided Iterative Policy
  Optimization method designed to gradually improve video LLMs' capacity for
  generating deep, long-term reasoning chains. The core component is
  Rollout-Aware Verifier, positioned between the GRPO and Direct Preference
  Optimization (DPO) training phases to form the GRPO-Verifier-DPO training loop.
  This verifier leverages small LLMs as a judge to assess the reasoning logic of
  rollouts, enabling the construction of high-quality contrastive data, including
  reflective and contextually consistent CoTs. These curated preference samples
  drive the efficient DPO stage (7x faster than GRPO), leading to marked
  improvements in reasoning chain quality, especially in terms of length and
  contextual consistency. This training loop benefits from GRPO's expansive
  search and DPO's targeted optimization. Experimental results demonstrate: 1)
  Significantly faster and more effective optimization compared to standard GRPO
  variants, yielding superior performance; 2) Our trained models exceed the
  direct inference of large-scale instruction-tuned Video-LLMs, producing long
  and contextually consistent CoTs on diverse video reasoning tasks; and 3) Our
  model with one iteration outperforms powerful LMMs (e.g., Kimi-VL) and long
  reasoning models (e.g., Video-R1), highlighting its effectiveness and
  stability.
paper_ai_summary: A Verifier-guided Iterative Policy Optimization method enhances Video-LLMs' reasoning capabilities by integrating a Rollout-Aware Verifier between GRPO and DPO phases, leading to faster and more effective optimization.
paper_ai_keywords: Reinforcement Learning Video Large Language Models Reinforcement Fine-Tuning Group Relative Policy Optimization Rollout-Aware Verifier Direct Preference Optimization long chain-of-thoughts video reasoning contrastive data reasoning chain quality contextual consistency