title: Self-alignment of Large Video Language Models with Refined Regularized
    Preference Optimization
summary: Despite recent advances in Large Video Language Models (LVLMs), they still
  struggle with fine-grained temporal understanding, hallucinate, and often make
  simple mistakes on even simple video question-answering tasks, all of which
  pose significant challenges to their safe and reliable deployment in real-world
  applications. To address these limitations, we propose a self-alignment
  framework that enables LVLMs to learn from their own errors. Our proposed
  framework first obtains a training set of preferred and non-preferred response
  pairs, where non-preferred responses are generated by incorporating common
  error patterns that often occur due to inadequate spatio-temporal
  understanding, spurious correlations between co-occurring concepts, and
  over-reliance on linguistic cues while neglecting the vision modality, among
  others. To facilitate self-alignment of LVLMs with the constructed preferred
  and non-preferred response pairs, we introduce Refined Regularized Preference
  Optimization (RRPO), a novel preference optimization method that utilizes
  sub-sequence-level refined rewards and token-wise KL regularization to address
  the limitations of Direct Preference Optimization (DPO). We demonstrate that
  RRPO achieves more precise alignment and more stable training compared to DPO.
  Our experiments and analysis validate the effectiveness of our approach across
  diverse video tasks, including video hallucination, short- and long-video
  understanding, and fine-grained temporal reasoning.
paper_authors._id: 68005c2d8da26a09080784c1
paper_authors.name: Pritam Sarkar
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-17T01:59:05.956Z
paper_authors._id: 6483b3d52193a1768c00c5ff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6483b3d52193a1768c00c5ff/uZ212A-sgMr5nAEvlEgHq.jpeg
paper_authors.fullname: Pritam Sarkar
paper_authors.type: user
paper_authors.user: pritamqu
paper_authors._id: 68005c2d8da26a09080784c2
paper_authors.name: Ali Etemad
paper_title: Self-alignment of Large Video Language Models with Refined Regularized
    Preference Optimization
paper_summary: Despite recent advances in Large Video Language Models (LVLMs), they still
  struggle with fine-grained temporal understanding, hallucinate, and often make
  simple mistakes on even simple video question-answering tasks, all of which
  pose significant challenges to their safe and reliable deployment in real-world
  applications. To address these limitations, we propose a self-alignment
  framework that enables LVLMs to learn from their own errors. Our proposed
  framework first obtains a training set of preferred and non-preferred response
  pairs, where non-preferred responses are generated by incorporating common
  error patterns that often occur due to inadequate spatio-temporal
  understanding, spurious correlations between co-occurring concepts, and
  over-reliance on linguistic cues while neglecting the vision modality, among
  others. To facilitate self-alignment of LVLMs with the constructed preferred
  and non-preferred response pairs, we introduce Refined Regularized Preference
  Optimization (RRPO), a novel preference optimization method that utilizes
  sub-sequence-level refined rewards and token-wise KL regularization to address
  the limitations of Direct Preference Optimization (DPO). We demonstrate that
  RRPO achieves more precise alignment and more stable training compared to DPO.
  Our experiments and analysis validate the effectiveness of our approach across
  diverse video tasks, including video hallucination, short- and long-video
  understanding, and fine-grained temporal reasoning.
paper_ai_summary: A self-alignment framework using Refined Regularized Preference Optimization helps Large Video Language Models improve accuracy and stability in video question-answering tasks.
paper_ai_keywords: Large Video Language Models self-alignment framework preferred and non-preferred response pairs inadequate spatio-temporal understanding spurious correlations linguistic cues Refined Regularized Preference Optimization sub-sequence-level refined rewards token-wise KL regularization Direct Preference Optimization video hallucination short-video understanding long-video understanding fine-grained temporal reasoning