title: ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of
    Text-to-Time-lapse Video Generation
summary: We propose a novel text-to-video (T2V) generation benchmark,
  ChronoMagic-Bench, to evaluate the temporal and metamorphic capabilities of the
  T2V models (e.g. Sora and Lumiere) in time-lapse video generation. In contrast
  to existing benchmarks that focus on the visual quality and textual relevance
  of generated videos, ChronoMagic-Bench focuses on the model's ability to
  generate time-lapse videos with significant metamorphic amplitude and temporal
  coherence. The benchmark probes T2V models for their physics, biology, and
  chemistry capabilities, in a free-form text query. For these purposes,
  ChronoMagic-Bench introduces 1,649 prompts and real-world videos as references,
  categorized into four major types of time-lapse videos: biological,
  human-created, meteorological, and physical phenomena, which are further
  divided into 75 subcategories. This categorization comprehensively evaluates
  the model's capacity to handle diverse and complex transformations. To
  accurately align human preference with the benchmark, we introduce two new
  automatic metrics, MTScore and CHScore, to evaluate the videos' metamorphic
  attributes and temporal coherence. MTScore measures the metamorphic amplitude,
  reflecting the degree of change over time, while CHScore assesses the temporal
  coherence, ensuring the generated videos maintain logical progression and
  continuity. Based on the ChronoMagic-Bench, we conduct comprehensive manual
  evaluations of ten representative T2V models, revealing their strengths and
  weaknesses across different categories of prompts, and providing a thorough
  evaluation framework that addresses current gaps in video generation research.
  Moreover, we create a large-scale ChronoMagic-Pro dataset, containing 460k
  high-quality pairs of 720p time-lapse videos and detailed captions ensuring
  high physical pertinence and large metamorphic amplitude.
paper_authors._id: 667d35b18549aa589d36a792
paper_authors.name: Shenghai Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:55:27.008Z
paper_authors._id: 63468720dd6d90d82ccf3450
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
paper_authors.fullname: YSH
paper_authors.type: user
paper_authors.user: BestWishYsh
paper_authors._id: 667d35b18549aa589d36a793
paper_authors.name: Jinfa Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:21:00.240Z
paper_authors._id: 63f37af60be81bdc5d92eebb
paper_authors.avatarUrl: /avatars/b8dfdff4ab36988ec9a8643e82a3d2db.svg
paper_authors.fullname: Huang
paper_authors.type: user
paper_authors.user: Jinfa
paper_authors._id: 667d35b18549aa589d36a794
paper_authors.name: Yongqi Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:21:06.428Z
paper_authors._id: 6614caef2da465c162b52abc
paper_authors.avatarUrl: /avatars/81b71cb400c01ac10bfac9d1a4f2a9fc.svg
paper_authors.fullname: Yongqi Xu
paper_authors.type: user
paper_authors.user: Cheliosoops
paper_authors._id: 667d35b18549aa589d36a795
paper_authors.name: Yaoyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:21:12.843Z
paper_authors._id: 662d968eac05b4f7c280d6f9
paper_authors.avatarUrl: /avatars/59d55142f33f650d9116a04110db839e.svg
paper_authors.fullname: YaoYang Liu
paper_authors.type: user
paper_authors.user: LazySheeep
paper_authors._id: 667d35b18549aa589d36a796
paper_authors.name: Shaofeng Zhang
paper_authors._id: 667d35b18549aa589d36a797
paper_authors.name: Yujun Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:34:46.735Z
paper_authors._id: 645ce309a03f3ebb0bdf2b89
paper_authors.avatarUrl: /avatars/652db4d2badd1191c6612f0950a88a88.svg
paper_authors.fullname: Yujun Shi
paper_authors.type: user
paper_authors.user: PeiyangShi
paper_authors._id: 667d35b18549aa589d36a798
paper_authors.name: Ruijie Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:15:59.458Z
paper_authors._id: 63ff09f24852102d4871c19c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ff09f24852102d4871c19c/lyE3xemtZss3qebK5sEXw.png
paper_authors.fullname: Rui-Jie Zhu
paper_authors.type: user
paper_authors.user: ridger
paper_authors._id: 667d35b18549aa589d36a799
paper_authors.name: Xinhua Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:35:27.216Z
paper_authors._id: 645b4026f9d4ec91fdd516ce
paper_authors.avatarUrl: /avatars/fb6556ab973764af5c44980f4e8fcc71.svg
paper_authors.fullname: Xinhua Cheng
paper_authors.type: user
paper_authors.user: cxh0519
paper_authors._id: 667d35b18549aa589d36a79a
paper_authors.name: Jiebo Luo
paper_authors._id: 667d35b18549aa589d36a79b
paper_authors.name: Li Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:11:57.415Z
paper_authors._id: 66135a5e50350afe76beebce
paper_authors.avatarUrl: /avatars/370a4b83949355feb050c2cb0425c264.svg
paper_authors.fullname: yl2488
paper_authors.type: user
paper_authors.user: yl2488
paper_title: ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of
    Text-to-Time-lapse Video Generation
paper_summary: We propose a novel text-to-video (T2V) generation benchmark,
  ChronoMagic-Bench, to evaluate the temporal and metamorphic capabilities of the
  T2V models (e.g. Sora and Lumiere) in time-lapse video generation. In contrast
  to existing benchmarks that focus on the visual quality and textual relevance
  of generated videos, ChronoMagic-Bench focuses on the model's ability to
  generate time-lapse videos with significant metamorphic amplitude and temporal
  coherence. The benchmark probes T2V models for their physics, biology, and
  chemistry capabilities, in a free-form text query. For these purposes,
  ChronoMagic-Bench introduces 1,649 prompts and real-world videos as references,
  categorized into four major types of time-lapse videos: biological,
  human-created, meteorological, and physical phenomena, which are further
  divided into 75 subcategories. This categorization comprehensively evaluates
  the model's capacity to handle diverse and complex transformations. To
  accurately align human preference with the benchmark, we introduce two new
  automatic metrics, MTScore and CHScore, to evaluate the videos' metamorphic
  attributes and temporal coherence. MTScore measures the metamorphic amplitude,
  reflecting the degree of change over time, while CHScore assesses the temporal
  coherence, ensuring the generated videos maintain logical progression and
  continuity. Based on the ChronoMagic-Bench, we conduct comprehensive manual
  evaluations of ten representative T2V models, revealing their strengths and
  weaknesses across different categories of prompts, and providing a thorough
  evaluation framework that addresses current gaps in video generation research.
  Moreover, we create a large-scale ChronoMagic-Pro dataset, containing 460k
  high-quality pairs of 720p time-lapse videos and detailed captions ensuring
  high physical pertinence and large metamorphic amplitude.
paper_ai_summary: ChronoMagic-Bench evaluates text-to-video models' ability to generate time-lapse videos with metamorphic amplitude and temporal coherence using novel metrics and a large dataset.
paper_ai_keywords: text-to-video generation time-lapse video generation metamorphic amplitude temporal coherence MTScore CHScore T2V models Sora Lumiere biological time-lapse human-created time-lapse meteorological time-lapse physical phenomena ChronoMagic-Pro dataset