title: Direct Preference Optimization of Video Large Multimodal Models from
    Language Model Reward
summary: Preference modeling techniques, such as direct preference optimization (DPO),
  has shown effective in enhancing the generalization abilities of large language
  model (LLM). However, in tasks involving video instruction-following, providing
  informative feedback, especially for detecting hallucinations in generated
  responses, remains a significant challenge. Previous studies have explored
  using large large multimodal models (LMMs) as reward models to guide preference
  modeling, but their ability to accurately assess the factuality of generated
  responses compared to corresponding videos has not been conclusively
  established. This paper introduces a novel framework that utilizes detailed
  video captions as a proxy of video content, enabling language models to
  incorporate this information as supporting evidence for scoring video Question
  Answering (QA) predictions. Our approach demonstrates robust alignment with
  OpenAI GPT-4V model's reward mechanism, which directly takes video frames as
  input. Furthermore, we show that applying this tailored reward through DPO
  significantly improves the performance of video LMMs on video QA tasks.
paper_authors._id: 660b94e2c30c49ec534b183d
paper_authors.name: Ruohong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-23T07:35:43.256Z
paper_authors._id: 63eeb0bbbe5d5900b85130ca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63eeb0bbbe5d5900b85130ca/ofmp91Zp7xAppQACCKz4L.jpeg
paper_authors.fullname: Ruohong Zhang
paper_authors.type: user
paper_authors.user: ruohongz
paper_authors._id: 660b94e2c30c49ec534b183e
paper_authors.name: Liangke Gui
paper_authors._id: 660b94e2c30c49ec534b183f
paper_authors.name: Zhiqing Sun
paper_authors._id: 660b94e2c30c49ec534b1840
paper_authors.name: Yihao Feng
paper_authors._id: 660b94e2c30c49ec534b1841
paper_authors.name: Keyang Xu
paper_authors._id: 660b94e2c30c49ec534b1842
paper_authors.name: Yuanhan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:24:12.774Z
paper_authors._id: 62a993d80472c0b7f94027df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
paper_authors.fullname: Zhang Yuanhan
paper_authors.type: user
paper_authors.user: ZhangYuanhan
paper_authors._id: 660b94e2c30c49ec534b1843
paper_authors.name: Di Fu
paper_authors._id: 660b94e2c30c49ec534b1844
paper_authors.name: Chunyuan Li
paper_authors._id: 660b94e2c30c49ec534b1845
paper_authors.name: Alexander Hauptmann
paper_authors._id: 660b94e2c30c49ec534b1846
paper_authors.name: Yonatan Bisk
paper_authors._id: 660b94e2c30c49ec534b1847
paper_authors.name: Yiming Yang
paper_title: Direct Preference Optimization of Video Large Multimodal Models from
    Language Model Reward
paper_summary: Preference modeling techniques, such as direct preference optimization (DPO),
  has shown effective in enhancing the generalization abilities of large language
  model (LLM). However, in tasks involving video instruction-following, providing
  informative feedback, especially for detecting hallucinations in generated
  responses, remains a significant challenge. Previous studies have explored
  using large large multimodal models (LMMs) as reward models to guide preference
  modeling, but their ability to accurately assess the factuality of generated
  responses compared to corresponding videos has not been conclusively
  established. This paper introduces a novel framework that utilizes detailed
  video captions as a proxy of video content, enabling language models to
  incorporate this information as supporting evidence for scoring video Question
  Answering (QA) predictions. Our approach demonstrates robust alignment with
  OpenAI GPT-4V model's reward mechanism, which directly takes video frames as
  input. Furthermore, we show that applying this tailored reward through DPO
  significantly improves the performance of video LMMs on video QA tasks.
paper_ai_keywords: direct preference optimization (DPO) large language model (LLM) video instruction-following hallucinations large multimodal models (LMMs) reward models detailed video captions video Question Answering (QA) OpenAI GPT-4V video frames