title: ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large
    Multimodal Models
summary: Latest developments in Large Multimodal Models (LMMs) have broadened their
  capabilities to include video understanding. Specifically, Text-to-video (T2V)
  models have made significant progress in quality, comprehension, and duration,
  excelling at creating videos from simple textual prompts. Yet, they still
  frequently produce hallucinated content that clearly signals the video is
  AI-generated. We introduce ViBe: a large-scale Text-to-Video Benchmark of
  hallucinated videos from T2V models. We identify five major types of
  hallucination: Vanishing Subject, Numeric Variability, Temporal Dysmorphia,
  Omission Error, and Physical Incongruity. Using 10 open-source T2V models, we
  developed the first large-scale dataset of hallucinated videos, comprising
  3,782 videos annotated by humans into these five categories. ViBe offers a
  unique resource for evaluating the reliability of T2V models and provides a
  foundation for improving hallucination detection and mitigation in video
  generation. We establish classification as a baseline and present various
  ensemble classifier configurations, with the TimeSFormer + CNN combination
  yielding the best performance, achieving 0.345 accuracy and 0.342 F1 score.
  This benchmark aims to drive the development of robust T2V models that produce
  videos more accurately aligned with input prompts.
paper_authors._id: 673f9c69502969a014facb0e
paper_authors.name: Vipula Rawte
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T10:02:07.037Z
paper_authors._id: 5e83bbbd0f1e020fa446aeb0
paper_authors.avatarUrl: /avatars/e627f98e84ce2f1c26005600a25c932c.svg
paper_authors.fullname: vr25
paper_authors.type: user
paper_authors.user: vr25
paper_authors._id: 673f9c69502969a014facb0f
paper_authors.name: Sarthak Jain
paper_authors._id: 673f9c69502969a014facb10
paper_authors.name: Aarush Sinha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-24T22:06:58.239Z
paper_authors._id: 64dca00a5f21fb7f85989628
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64dca00a5f21fb7f85989628/rJx4lxhTr4vkmdSkDuSYi.jpeg
paper_authors.fullname: Aarush
paper_authors.type: user
paper_authors.user: chungimungi
paper_authors._id: 673f9c69502969a014facb11
paper_authors.name: Garv Kaushik
paper_authors._id: 673f9c69502969a014facb12
paper_authors.name: Aman Bansal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-24T09:08:25.944Z
paper_authors._id: 65c278d447ac0454b612ebdb
paper_authors.avatarUrl: /avatars/63636a78b7ba289260e112603e4ee63c.svg
paper_authors.fullname: Aman Bansal
paper_authors.type: user
paper_authors.user: bansalaman18
paper_authors._id: 673f9c69502969a014facb13
paper_authors.name: Prathiksha Rumale Vishwanath
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-24T22:07:01.097Z
paper_authors._id: 667247ae09651d9c0078b536
paper_authors.avatarUrl: /avatars/462275c1267e8b04a45559e21d59bd73.svg
paper_authors.fullname: Prathiksha Rumale Vishwanath
paper_authors.type: user
paper_authors.user: prathiksha-rv
paper_authors._id: 673f9c69502969a014facb14
paper_authors.name: Samyak Rajesh Jain
paper_authors._id: 673f9c69502969a014facb15
paper_authors.name: Aishwarya Naresh Reganti
paper_authors._id: 673f9c69502969a014facb16
paper_authors.name: Vinija Jain
paper_authors._id: 673f9c69502969a014facb17
paper_authors.name: Aman Chadha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T21:32:42.577Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_authors._id: 673f9c69502969a014facb18
paper_authors.name: Amit P. Sheth
paper_authors._id: 673f9c69502969a014facb19
paper_authors.name: Amitava Das
paper_title: ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large
    Multimodal Models
paper_summary: Latest developments in Large Multimodal Models (LMMs) have broadened their
  capabilities to include video understanding. Specifically, Text-to-video (T2V)
  models have made significant progress in quality, comprehension, and duration,
  excelling at creating videos from simple textual prompts. Yet, they still
  frequently produce hallucinated content that clearly signals the video is
  AI-generated. We introduce ViBe: a large-scale Text-to-Video Benchmark of
  hallucinated videos from T2V models. We identify five major types of
  hallucination: Vanishing Subject, Numeric Variability, Temporal Dysmorphia,
  Omission Error, and Physical Incongruity. Using 10 open-source T2V models, we
  developed the first large-scale dataset of hallucinated videos, comprising
  3,782 videos annotated by humans into these five categories. ViBe offers a
  unique resource for evaluating the reliability of T2V models and provides a
  foundation for improving hallucination detection and mitigation in video
  generation. We establish classification as a baseline and present various
  ensemble classifier configurations, with the TimeSFormer + CNN combination
  yielding the best performance, achieving 0.345 accuracy and 0.342 F1 score.
  This benchmark aims to drive the development of robust T2V models that produce
  videos more accurately aligned with input prompts.
paper_ai_keywords: Large Multimodal Models (LMMs) Text-to-video (T2V) models Vanishing Subject Numeric Variability Temporal Dysmorphia Omission Error Physical Incongruity TimeSFormer CNN (Convolutional Neural Network) Accuracy F1 score