title: VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in
    Large Video-Language Models
summary: Recent advancements in Multimodal Large Language Models (MLLMs) have extended
  their capabilities to video understanding. Yet, these models are often plagued
  by "hallucinations", where irrelevant or nonsensical content is generated,
  deviating from the actual video context. This work introduces VideoHallucer,
  the first comprehensive benchmark for hallucination detection in large
  video-language models (LVLMs). VideoHallucer categorizes hallucinations into
  two main types: intrinsic and extrinsic, offering further subcategories for
  detailed analysis, including object-relation, temporal, semantic detail,
  extrinsic factual, and extrinsic non-factual hallucinations. We adopt an
  adversarial binary VideoQA method for comprehensive evaluation, where pairs of
  basic and hallucinated questions are crafted strategically. By evaluating
  eleven LVLMs on VideoHallucer, we reveal that i) the majority of current models
  exhibit significant issues with hallucinations; ii) while scaling datasets and
  parameters improves models' ability to detect basic visual cues and
  counterfactuals, it provides limited benefit for detecting extrinsic factual
  hallucinations; iii) existing models are more adept at detecting facts than
  identifying hallucinations. As a byproduct, these analyses further instruct the
  development of our self-PEP framework, achieving an average of 5.38%
  improvement in hallucination resistance across all model architectures.
paper_authors._id: 667a2f6cb2401c64d7e0169c
paper_authors.name: Yuxuan Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-25T05:47:11.864Z
paper_authors._id: 60b9e6837946aff342f734ae
paper_authors.avatarUrl: /avatars/a711a6aa35757dfd7b78b26098a964fc.svg
paper_authors.fullname: Yuxuan Wang
paper_authors.type: user
paper_authors.user: ColorfulAI
paper_authors._id: 667a2f6cb2401c64d7e0169d
paper_authors.name: Yueqian Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:14:29.905Z
paper_authors._id: 643672220b2c0d86d0b0f639
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643672220b2c0d86d0b0f639/8nqZku8HxsSL_zR3XJPqK.png
paper_authors.fullname: Yueqian Wang
paper_authors.type: user
paper_authors.user: wangyueqian
paper_authors._id: 667a2f6cb2401c64d7e0169e
paper_authors.name: Dongyan Zhao
paper_authors._id: 667a2f6cb2401c64d7e0169f
paper_authors.name: Cihang Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:14:51.184Z
paper_authors._id: 645eb61da3c5cd8a16efffff
paper_authors.avatarUrl: /avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
paper_authors.fullname: Cihang Xie
paper_authors.type: user
paper_authors.user: cihangxie
paper_authors._id: 667a2f6cb2401c64d7e016a0
paper_authors.name: Zilong Zheng
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-25T02:46:08.771Z
paper_authors._id: 63a95a6a7930fa8c7dd63d4e
paper_authors.avatarUrl: /avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
paper_authors.fullname: Zilong Zheng
paper_authors.type: user
paper_authors.user: zlzheng
paper_title: VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in
    Large Video-Language Models
paper_summary: Recent advancements in Multimodal Large Language Models (MLLMs) have extended
  their capabilities to video understanding. Yet, these models are often plagued
  by "hallucinations", where irrelevant or nonsensical content is generated,
  deviating from the actual video context. This work introduces VideoHallucer,
  the first comprehensive benchmark for hallucination detection in large
  video-language models (LVLMs). VideoHallucer categorizes hallucinations into
  two main types: intrinsic and extrinsic, offering further subcategories for
  detailed analysis, including object-relation, temporal, semantic detail,
  extrinsic factual, and extrinsic non-factual hallucinations. We adopt an
  adversarial binary VideoQA method for comprehensive evaluation, where pairs of
  basic and hallucinated questions are crafted strategically. By evaluating
  eleven LVLMs on VideoHallucer, we reveal that i) the majority of current models
  exhibit significant issues with hallucinations; ii) while scaling datasets and
  parameters improves models' ability to detect basic visual cues and
  counterfactuals, it provides limited benefit for detecting extrinsic factual
  hallucinations; iii) existing models are more adept at detecting facts than
  identifying hallucinations. As a byproduct, these analyses further instruct the
  development of our self-PEP framework, achieving an average of 5.38%
  improvement in hallucination resistance across all model architectures.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) video understanding hallucinations VideoHallucer large video-language models (LVLMs) intrinsic hallucinations extrinsic hallucinations object-relation hallucinations temporal hallucinations semantic detail hallucinations extrinsic factual hallucinations extrinsic non-factual hallucinations Adversarial BinaryVideoQA self-PEP framework hallucination resistance