title: Impossible Videos
summary: Synthetic videos nowadays is widely used to complement data scarcity and
  diversity of real-world videos. Current synthetic datasets primarily replicate
  real-world scenarios, leaving impossible, counterfactual and anti-reality video
  concepts underexplored. This work aims to answer two questions: 1) Can today's
  video generation models effectively follow prompts to create impossible video
  content? 2) Are today's video understanding models good enough for
  understanding impossible videos? To this end, we introduce IPV-Bench, a novel
  benchmark designed to evaluate and foster progress in video understanding and
  generation. IPV-Bench is underpinned by a comprehensive taxonomy, encompassing
  4 domains, 14 categories. It features diverse scenes that defy physical,
  biological, geographical, or social laws. Based on the taxonomy, a prompt suite
  is constructed to evaluate video generation models, challenging their prompt
  following and creativity capabilities. In addition, a video benchmark is
  curated to assess Video-LLMs on their ability of understanding impossible
  videos, which particularly requires reasoning on temporal dynamics and world
  knowledge. Comprehensive evaluations reveal limitations and insights for future
  directions of video models, paving the way for next-generation video models.
paper_authors._id: 67da1ee1f1a4a52e8a1e0241
paper_authors.name: Zechen Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:45:09.659Z
paper_authors._id: 64b7833aa5018e3c7c9b50d8
paper_authors.avatarUrl: /avatars/782415605ed786b73f484fcc86a6384f.svg
paper_authors.fullname: Zechen Bai
paper_authors.type: user
paper_authors.user: ZechenBai
paper_authors._id: 67da1ee1f1a4a52e8a1e0242
paper_authors.name: Hai Ci
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:14:19.533Z
paper_authors._id: 647352516a972f252de1fd58
paper_authors.avatarUrl: /avatars/8e93ca66e01f047c5fb28e1c4f737e8e.svg
paper_authors.fullname: Hai Ci
paper_authors.type: user
paper_authors.user: HaiCi
paper_authors._id: 67da1ee1f1a4a52e8a1e0243
paper_authors.name: Mike Zheng Shou
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-19T01:33:23.736Z
paper_authors._id: 63a55320ce5763e06f78519c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671779060549-noauth.jpeg
paper_authors.fullname: Mike Shou
paper_authors.type: user
paper_authors.user: mikeshou
paper_title: Impossible Videos
paper_summary: Synthetic videos nowadays is widely used to complement data scarcity and
  diversity of real-world videos. Current synthetic datasets primarily replicate
  real-world scenarios, leaving impossible, counterfactual and anti-reality video
  concepts underexplored. This work aims to answer two questions: 1) Can today's
  video generation models effectively follow prompts to create impossible video
  content? 2) Are today's video understanding models good enough for
  understanding impossible videos? To this end, we introduce IPV-Bench, a novel
  benchmark designed to evaluate and foster progress in video understanding and
  generation. IPV-Bench is underpinned by a comprehensive taxonomy, encompassing
  4 domains, 14 categories. It features diverse scenes that defy physical,
  biological, geographical, or social laws. Based on the taxonomy, a prompt suite
  is constructed to evaluate video generation models, challenging their prompt
  following and creativity capabilities. In addition, a video benchmark is
  curated to assess Video-LLMs on their ability of understanding impossible
  videos, which particularly requires reasoning on temporal dynamics and world
  knowledge. Comprehensive evaluations reveal limitations and insights for future
  directions of video models, paving the way for next-generation video models.
paper_ai_summary: IPV-Bench evaluates video generation and understanding models on creating and interpreting impossible videos, highlighting their limitations and guiding future advancements.
paper_ai_keywords: video generation models prompt following creativity video understanding models temporal dynamics world knowledge video-LLMs IPV-Bench