EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
The vision and language generative models have been overgrown in recent
years. For video generation, various open-sourced models and public-available
services are released for generating high-visual quality videos. However, these
methods often use a few academic metrics, for example, FVD or IS, to evaluate
the performance. We argue that it is hard to judge the large conditional
generative models from the simple metrics since these models are often trained
on very large datasets with multi-aspect abilities. Thus, we propose a new
framework and pipeline to exhaustively evaluate the performance of the
generated videos. To achieve this, we first conduct a new prompt list for
text-to-video generation by analyzing the real-world prompt list with the help
of the large language model. Then, we evaluate the state-of-the-art video
generative models on our carefully designed benchmarks, in terms of visual
qualities, content qualities, motion qualities, and text-caption alignment with
around 18 objective metrics. To obtain the final leaderboard of the models, we
also fit a series of coefficients to align the objective metrics to the users'
opinions. Based on the proposed opinion alignment method, our final score shows
a higher correlation than simply averaging the metrics, showing the
effectiveness of the proposed evaluation method.
652f507a902fe76a6d22b884
Yaofang Liu
claimed_verified
2024-01-23T10:35:36.213Z
658d46649a1397992a4d657f
/avatars/66d23095b8d0368ea862778dbf123a67.svg
Raphael Liu
user
RaphaelLiu
652f507a902fe76a6d22b885
Xiaodong Cun
admin_assigned
2023-10-18T09:13:30.459Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
652f507a902fe76a6d22b886
Xuebo Liu
claimed_verified
2023-10-18T13:42:54.686Z
652fbf9e75f5f7a84e10cb2e
/avatars/73e7af465dbb57826a52e49c3e72f55f.svg
Xuebo Liu
user
SunbowLiu
652f507a902fe76a6d22b887
Xintao Wang
admin_assigned
2023-10-18T09:14:05.495Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
652f507a902fe76a6d22b888
Yong Zhang
652f507a902fe76a6d22b889
Haoxin Chen
claimed_verified
2024-01-19T09:23:52.103Z
6305cb2f435ec751b723d80a
/avatars/5e8519d9814038726bf98f4104f74568.svg
Haoxin Chen
user
paulai
652f507a902fe76a6d22b88a
Yang Liu
652f507a902fe76a6d22b88b
Tieyong Zeng
652f507a902fe76a6d22b88c
Raymond Chan
652f507a902fe76a6d22b88d
Ying Shan
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
The vision and language generative models have been overgrown in recent
years. For video generation, various open-sourced models and public-available
services are released for generating high-visual quality videos. However, these
methods often use a few academic metrics, for example, FVD or IS, to evaluate
the performance. We argue that it is hard to judge the large conditional
generative models from the simple metrics since these models are often trained
on very large datasets with multi-aspect abilities. Thus, we propose a new
framework and pipeline to exhaustively evaluate the performance of the
generated videos. To achieve this, we first conduct a new prompt list for
text-to-video generation by analyzing the real-world prompt list with the help
of the large language model. Then, we evaluate the state-of-the-art video
generative models on our carefully designed benchmarks, in terms of visual
qualities, content qualities, motion qualities, and text-caption alignment with
around 18 objective metrics. To obtain the final leaderboard of the models, we
also fit a series of coefficients to align the objective metrics to the users'
opinions. Based on the proposed opinion alignment method, our final score shows
a higher correlation than simply averaging the metrics, showing the
effectiveness of the proposed evaluation method.
A new framework evaluates vision-language generative models for video creation by using a nuanced benchmark across visual, content, motion, and textual quality metrics, aligned with user opinions.
vision and language generative models text-to-video generation large language model video generative models prompt list objective metrics opinion alignment method