ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large
  Multimodal Models
Latest developments in Large Multimodal Models (LMMs) have broadened their
capabilities to include video understanding. Specifically, Text-to-video (T2V)
models have made significant progress in quality, comprehension, and duration,
excelling at creating videos from simple textual prompts. Yet, they still
frequently produce hallucinated content that clearly signals the video is
AI-generated. We introduce ViBe: a large-scale Text-to-Video Benchmark of
hallucinated videos from T2V models. We identify five major types of
hallucination: Vanishing Subject, Numeric Variability, Temporal Dysmorphia,
Omission Error, and Physical Incongruity. Using 10 open-source T2V models, we
developed the first large-scale dataset of hallucinated videos, comprising
3,782 videos annotated by humans into these five categories. ViBe offers a
unique resource for evaluating the reliability of T2V models and provides a
foundation for improving hallucination detection and mitigation in video
generation. We establish classification as a baseline and present various
ensemble classifier configurations, with the TimeSFormer + CNN combination
yielding the best performance, achieving 0.345 accuracy and 0.342 F1 score.
This benchmark aims to drive the development of robust T2V models that produce
videos more accurately aligned with input prompts.
673f9c69502969a014facb0e
Vipula Rawte
claimed_verified
2024-11-22T10:02:07.037Z
5e83bbbd0f1e020fa446aeb0
/avatars/e627f98e84ce2f1c26005600a25c932c.svg
vr25
user
vr25
673f9c69502969a014facb0f
Sarthak Jain
673f9c69502969a014facb10
Aarush Sinha
claimed_verified
2024-11-24T22:06:58.239Z
64dca00a5f21fb7f85989628
https://cdn-avatars.huggingface.co/v1/production/uploads/64dca00a5f21fb7f85989628/rJx4lxhTr4vkmdSkDuSYi.jpeg
Aarush
user
chungimungi
673f9c69502969a014facb11
Garv Kaushik
673f9c69502969a014facb12
Aman Bansal
claimed_verified
2025-01-24T09:08:25.944Z
65c278d447ac0454b612ebdb
/avatars/63636a78b7ba289260e112603e4ee63c.svg
Aman Bansal
user
bansalaman18
673f9c69502969a014facb13
Prathiksha Rumale Vishwanath
claimed_verified
2024-11-24T22:07:01.097Z
667247ae09651d9c0078b536
/avatars/462275c1267e8b04a45559e21d59bd73.svg
Prathiksha Rumale Vishwanath
user
prathiksha-rv
673f9c69502969a014facb14
Samyak Rajesh Jain
673f9c69502969a014facb15
Aishwarya Naresh Reganti
673f9c69502969a014facb16
Vinija Jain
673f9c69502969a014facb17
Aman Chadha
claimed_verified
2024-11-21T21:32:42.577Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
673f9c69502969a014facb18
Amit P. Sheth
673f9c69502969a014facb19
Amitava Das
ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large
  Multimodal Models
Latest developments in Large Multimodal Models (LMMs) have broadened their
capabilities to include video understanding. Specifically, Text-to-video (T2V)
models have made significant progress in quality, comprehension, and duration,
excelling at creating videos from simple textual prompts. Yet, they still
frequently produce hallucinated content that clearly signals the video is
AI-generated. We introduce ViBe: a large-scale Text-to-Video Benchmark of
hallucinated videos from T2V models. We identify five major types of
hallucination: Vanishing Subject, Numeric Variability, Temporal Dysmorphia,
Omission Error, and Physical Incongruity. Using 10 open-source T2V models, we
developed the first large-scale dataset of hallucinated videos, comprising
3,782 videos annotated by humans into these five categories. ViBe offers a
unique resource for evaluating the reliability of T2V models and provides a
foundation for improving hallucination detection and mitigation in video
generation. We establish classification as a baseline and present various
ensemble classifier configurations, with the TimeSFormer + CNN combination
yielding the best performance, achieving 0.345 accuracy and 0.342 F1 score.
This benchmark aims to drive the development of robust T2V models that produce
videos more accurately aligned with input prompts.
ViBe is a large-scale Text-to-Video Benchmark for evaluating hallucination in video generation with annotations for five types of hallucination, using ensemble classifiers to improve detection and mitigation.
Large Multimodal Models Text-to-video hallucinated content TimeSFormer CNN classification ensemble classifier configurations accuracy F1 score