title: EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
summary: The vision and language generative models have been overgrown in recent
  years. For video generation, various open-sourced models and public-available
  services are released for generating high-visual quality videos. However, these
  methods often use a few academic metrics, for example, FVD or IS, to evaluate
  the performance. We argue that it is hard to judge the large conditional
  generative models from the simple metrics since these models are often trained
  on very large datasets with multi-aspect abilities. Thus, we propose a new
  framework and pipeline to exhaustively evaluate the performance of the
  generated videos. To achieve this, we first conduct a new prompt list for
  text-to-video generation by analyzing the real-world prompt list with the help
  of the large language model. Then, we evaluate the state-of-the-art video
  generative models on our carefully designed benchmarks, in terms of visual
  qualities, content qualities, motion qualities, and text-caption alignment with
  around 18 objective metrics. To obtain the final leaderboard of the models, we
  also fit a series of coefficients to align the objective metrics to the users'
  opinions. Based on the proposed opinion alignment method, our final score shows
  a higher correlation than simply averaging the metrics, showing the
  effectiveness of the proposed evaluation method.
paper_authors._id: 652f507a902fe76a6d22b884
paper_authors.name: Yaofang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T10:35:36.213Z
paper_authors._id: 658d46649a1397992a4d657f
paper_authors.avatarUrl: /avatars/66d23095b8d0368ea862778dbf123a67.svg
paper_authors.fullname: Raphael Liu
paper_authors.type: user
paper_authors.user: RaphaelLiu
paper_authors._id: 652f507a902fe76a6d22b885
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:13:30.459Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 652f507a902fe76a6d22b886
paper_authors.name: Xuebo Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-18T13:42:54.686Z
paper_authors._id: 652fbf9e75f5f7a84e10cb2e
paper_authors.avatarUrl: /avatars/73e7af465dbb57826a52e49c3e72f55f.svg
paper_authors.fullname: Xuebo Liu
paper_authors.type: user
paper_authors.user: SunbowLiu
paper_authors._id: 652f507a902fe76a6d22b887
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:14:05.495Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 652f507a902fe76a6d22b888
paper_authors.name: Yong Zhang
paper_authors._id: 652f507a902fe76a6d22b889
paper_authors.name: Haoxin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-19T09:23:52.103Z
paper_authors._id: 6305cb2f435ec751b723d80a
paper_authors.avatarUrl: /avatars/5e8519d9814038726bf98f4104f74568.svg
paper_authors.fullname: Haoxin Chen
paper_authors.type: user
paper_authors.user: paulai
paper_authors._id: 652f507a902fe76a6d22b88a
paper_authors.name: Yang Liu
paper_authors._id: 652f507a902fe76a6d22b88b
paper_authors.name: Tieyong Zeng
paper_authors._id: 652f507a902fe76a6d22b88c
paper_authors.name: Raymond Chan
paper_authors._id: 652f507a902fe76a6d22b88d
paper_authors.name: Ying Shan
paper_title: EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
paper_summary: The vision and language generative models have been overgrown in recent
  years. For video generation, various open-sourced models and public-available
  services are released for generating high-visual quality videos. However, these
  methods often use a few academic metrics, for example, FVD or IS, to evaluate
  the performance. We argue that it is hard to judge the large conditional
  generative models from the simple metrics since these models are often trained
  on very large datasets with multi-aspect abilities. Thus, we propose a new
  framework and pipeline to exhaustively evaluate the performance of the
  generated videos. To achieve this, we first conduct a new prompt list for
  text-to-video generation by analyzing the real-world prompt list with the help
  of the large language model. Then, we evaluate the state-of-the-art video
  generative models on our carefully designed benchmarks, in terms of visual
  qualities, content qualities, motion qualities, and text-caption alignment with
  around 18 objective metrics. To obtain the final leaderboard of the models, we
  also fit a series of coefficients to align the objective metrics to the users'
  opinions. Based on the proposed opinion alignment method, our final score shows
  a higher correlation than simply averaging the metrics, showing the
  effectiveness of the proposed evaluation method.
paper_ai_keywords: vision and language generative models video generation FVD IS large conditional generative models prompt list text-to-video generation large language model visual qualities content qualities motion qualities text-caption alignment objective metrics opinion alignment method