ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of
  Text-to-Time-lapse Video Generation
We propose a novel text-to-video (T2V) generation benchmark,
ChronoMagic-Bench, to evaluate the temporal and metamorphic capabilities of the
T2V models (e.g. Sora and Lumiere) in time-lapse video generation. In contrast
to existing benchmarks that focus on the visual quality and textual relevance
of generated videos, ChronoMagic-Bench focuses on the model's ability to
generate time-lapse videos with significant metamorphic amplitude and temporal
coherence. The benchmark probes T2V models for their physics, biology, and
chemistry capabilities, in a free-form text query. For these purposes,
ChronoMagic-Bench introduces 1,649 prompts and real-world videos as references,
categorized into four major types of time-lapse videos: biological,
human-created, meteorological, and physical phenomena, which are further
divided into 75 subcategories. This categorization comprehensively evaluates
the model's capacity to handle diverse and complex transformations. To
accurately align human preference with the benchmark, we introduce two new
automatic metrics, MTScore and CHScore, to evaluate the videos' metamorphic
attributes and temporal coherence. MTScore measures the metamorphic amplitude,
reflecting the degree of change over time, while CHScore assesses the temporal
coherence, ensuring the generated videos maintain logical progression and
continuity. Based on the ChronoMagic-Bench, we conduct comprehensive manual
evaluations of ten representative T2V models, revealing their strengths and
weaknesses across different categories of prompts, and providing a thorough
evaluation framework that addresses current gaps in video generation research.
Moreover, we create a large-scale ChronoMagic-Pro dataset, containing 460k
high-quality pairs of 720p time-lapse videos and detailed captions ensuring
high physical pertinence and large metamorphic amplitude.
667d35b18549aa589d36a792
Shenghai Yuan
claimed_verified
2024-06-27T09:55:27.008Z
63468720dd6d90d82ccf3450
https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
YSH
user
BestWishYsh
667d35b18549aa589d36a793
Jinfa Huang
admin_assigned
2024-06-27T18:21:00.240Z
63f37af60be81bdc5d92eebb
/avatars/b8dfdff4ab36988ec9a8643e82a3d2db.svg
Huang
user
Jinfa
667d35b18549aa589d36a794
Yongqi Xu
admin_assigned
2024-06-27T18:21:06.428Z
6614caef2da465c162b52abc
/avatars/81b71cb400c01ac10bfac9d1a4f2a9fc.svg
Yongqi Xu
user
Cheliosoops
667d35b18549aa589d36a795
Yaoyang Liu
admin_assigned
2024-06-27T18:21:12.843Z
662d968eac05b4f7c280d6f9
/avatars/59d55142f33f650d9116a04110db839e.svg
YaoYang Liu
user
LazySheeep
667d35b18549aa589d36a796
Shaofeng Zhang
667d35b18549aa589d36a797
Yujun Shi
admin_assigned
2024-06-27T18:34:46.735Z
645ce309a03f3ebb0bdf2b89
/avatars/652db4d2badd1191c6612f0950a88a88.svg
Yujun Shi
user
PeiyangShi
667d35b18549aa589d36a798
Ruijie Zhu
claimed_verified
2024-10-14T19:15:59.458Z
63ff09f24852102d4871c19c
https://cdn-avatars.huggingface.co/v1/production/uploads/63ff09f24852102d4871c19c/lyE3xemtZss3qebK5sEXw.png
Rui-Jie Zhu
user
ridger
667d35b18549aa589d36a799
Xinhua Cheng
admin_assigned
2024-06-27T18:35:27.216Z
645b4026f9d4ec91fdd516ce
/avatars/fb6556ab973764af5c44980f4e8fcc71.svg
Xinhua Cheng
user
cxh0519
667d35b18549aa589d36a79a
Jiebo Luo
667d35b18549aa589d36a79b
Li Yuan
claimed_verified
2025-04-04T07:11:57.415Z
66135a5e50350afe76beebce
/avatars/370a4b83949355feb050c2cb0425c264.svg
yl2488
user
yl2488
ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of
  Text-to-Time-lapse Video Generation
We propose a novel text-to-video (T2V) generation benchmark,
ChronoMagic-Bench, to evaluate the temporal and metamorphic capabilities of the
T2V models (e.g. Sora and Lumiere) in time-lapse video generation. In contrast
to existing benchmarks that focus on the visual quality and textual relevance
of generated videos, ChronoMagic-Bench focuses on the model's ability to
generate time-lapse videos with significant metamorphic amplitude and temporal
coherence. The benchmark probes T2V models for their physics, biology, and
chemistry capabilities, in a free-form text query. For these purposes,
ChronoMagic-Bench introduces 1,649 prompts and real-world videos as references,
categorized into four major types of time-lapse videos: biological,
human-created, meteorological, and physical phenomena, which are further
divided into 75 subcategories. This categorization comprehensively evaluates
the model's capacity to handle diverse and complex transformations. To
accurately align human preference with the benchmark, we introduce two new
automatic metrics, MTScore and CHScore, to evaluate the videos' metamorphic
attributes and temporal coherence. MTScore measures the metamorphic amplitude,
reflecting the degree of change over time, while CHScore assesses the temporal
coherence, ensuring the generated videos maintain logical progression and
continuity. Based on the ChronoMagic-Bench, we conduct comprehensive manual
evaluations of ten representative T2V models, revealing their strengths and
weaknesses across different categories of prompts, and providing a thorough
evaluation framework that addresses current gaps in video generation research.
Moreover, we create a large-scale ChronoMagic-Pro dataset, containing 460k
high-quality pairs of 720p time-lapse videos and detailed captions ensuring
high physical pertinence and large metamorphic amplitude.
ChronoMagic-Bench evaluates text-to-video models' ability to generate time-lapse videos with metamorphic amplitude and temporal coherence using novel metrics and a large dataset.
text-to-video generation time-lapse video generation metamorphic amplitude temporal coherence MTScore CHScore T2V models Sora Lumiere biological time-lapse human-created time-lapse meteorological time-lapse physical phenomena ChronoMagic-Pro dataset