title: TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and
    Image-to-Video Generation
summary: Video generation has many unique challenges beyond those of image generation.
  The temporal dimension introduces extensive possible variations across frames,
  over which consistency and continuity may be violated. In this study, we move
  beyond evaluating simple actions and argue that generated videos should
  incorporate the emergence of new concepts and their relation transitions like
  in real-world videos as time progresses. To assess the Temporal
  Compositionality of video generation models, we propose TC-Bench, a benchmark
  of meticulously crafted text prompts, corresponding ground truth videos, and
  robust evaluation metrics. The prompts articulate the initial and final states
  of scenes, effectively reducing ambiguities for frame development and
  simplifying the assessment of transition completion. In addition, by collecting
  aligned real-world videos corresponding to the prompts, we expand TC-Bench's
  applicability from text-conditional models to image-conditional ones that can
  perform generative frame interpolation. We also develop new metrics to measure
  the completeness of component transitions in generated videos, which
  demonstrate significantly higher correlations with human judgments than
  existing metrics. Our comprehensive experimental results reveal that most video
  generators achieve less than 20% of the compositional changes, highlighting
  enormous space for future improvement. Our analysis indicates that current
  video generation models struggle to interpret descriptions of compositional
  changes and synthesize various components across different time steps.
paper_authors._id: 666bbcbcd85fde8ee602af1d
paper_authors.name: Weixi Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:22:19.259Z
paper_authors._id: 6259e9489326c60347dd2b6c
paper_authors.avatarUrl: /avatars/6aae6282ddee96b899b7384154e9b0a0.svg
paper_authors.fullname: Weixi Feng
paper_authors.type: user
paper_authors.user: weixifeng
paper_authors._id: 666bbcbcd85fde8ee602af1e
paper_authors.name: Jiachen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:20:13.124Z
paper_authors._id: 640d7ec5fdeaae13907fc488
paper_authors.avatarUrl: /avatars/acf43ea155105a51c8612dacc4725091.svg
paper_authors.fullname: Jiachen Li
paper_authors.type: user
paper_authors.user: jiachenli-ucsb
paper_authors._id: 666bbcbcd85fde8ee602af1f
paper_authors.name: Michael Saxon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-08T20:00:58.657Z
paper_authors._id: 6002c1db698168af3bb9f4a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1610793426015-noauth.png
paper_authors.fullname: Michael Saxon
paper_authors.type: user
paper_authors.user: saxon
paper_authors._id: 666bbcbcd85fde8ee602af20
paper_authors.name: Tsu-jui Fu
paper_authors._id: 666bbcbcd85fde8ee602af21
paper_authors.name: Wenhu Chen
paper_authors._id: 666bbcbcd85fde8ee602af22
paper_authors.name: William Yang Wang
paper_title: TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and
    Image-to-Video Generation
paper_summary: Video generation has many unique challenges beyond those of image generation.
  The temporal dimension introduces extensive possible variations across frames,
  over which consistency and continuity may be violated. In this study, we move
  beyond evaluating simple actions and argue that generated videos should
  incorporate the emergence of new concepts and their relation transitions like
  in real-world videos as time progresses. To assess the Temporal
  Compositionality of video generation models, we propose TC-Bench, a benchmark
  of meticulously crafted text prompts, corresponding ground truth videos, and
  robust evaluation metrics. The prompts articulate the initial and final states
  of scenes, effectively reducing ambiguities for frame development and
  simplifying the assessment of transition completion. In addition, by collecting
  aligned real-world videos corresponding to the prompts, we expand TC-Bench's
  applicability from text-conditional models to image-conditional ones that can
  perform generative frame interpolation. We also develop new metrics to measure
  the completeness of component transitions in generated videos, which
  demonstrate significantly higher correlations with human judgments than
  existing metrics. Our comprehensive experimental results reveal that most video
  generators achieve less than 20% of the compositional changes, highlighting
  enormous space for future improvement. Our analysis indicates that current
  video generation models struggle to interpret descriptions of compositional
  changes and synthesize various components across different time steps.
paper_ai_summary: A benchmark named TC-Bench is proposed to evaluate the Temporal Compositionality of video generation models, highlighting significant gaps in current models' ability to handle compositional changes and transitions over time.
paper_ai_keywords: TC-Bench Temporal Compositionality text prompts ground truth videos evaluation metrics generative frame interpolation frame development transition completion human judgments compositional changes