title: VBench++: Comprehensive and Versatile Benchmark Suite for Video
    Generative Models
summary: Video generation has witnessed significant advancements, yet evaluating these
  models remains a challenge. A comprehensive evaluation benchmark for video
  generation is indispensable for two reasons: 1) Existing metrics do not fully
  align with human perceptions; 2) An ideal evaluation system should provide
  insights to inform future developments of video generation. To this end, we
  present VBench, a comprehensive benchmark suite that dissects "video generation
  quality" into specific, hierarchical, and disentangled dimensions, each with
  tailored prompts and evaluation methods. VBench has several appealing
  properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in
  video generation (e.g., subject identity inconsistency, motion smoothness,
  temporal flickering, and spatial relationship, etc). The evaluation metrics
  with fine-grained levels reveal individual models' strengths and weaknesses. 2)
  Human Alignment: We also provide a dataset of human preference annotations to
  validate our benchmarks' alignment with human perception, for each evaluation
  dimension respectively. 3) Valuable Insights: We look into current models'
  ability across various evaluation dimensions, and various content types. We
  also investigate the gaps between video and image generation models. 4)
  Versatile Benchmarking: VBench++ supports evaluating text-to-video and
  image-to-video. We introduce a high-quality Image Suite with an adaptive aspect
  ratio to enable fair evaluations across different image-to-video generation
  settings. Beyond assessing technical quality, VBench++ evaluates the
  trustworthiness of video generative models, providing a more holistic view of
  model performance. 5) Full Open-Sourcing: We fully open-source VBench++ and
  continually add new video generation models to our leaderboard to drive forward
  the field of video generation.
paper_authors._id: 673ecb860ebca74a17b0726f
paper_authors.name: Ziqi Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:24:09.580Z
paper_authors._id: 60efe7fa0d920bc7805cada5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png
paper_authors.fullname: Ziqi Huang
paper_authors.type: user
paper_authors.user: Ziqi
paper_authors._id: 673ecb860ebca74a17b07270
paper_authors.name: Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:04:37.850Z
paper_authors._id: 61f24cbb88b9b5abbe184a85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61f24cbb88b9b5abbe184a85/OvcJRU51yI8pdO77NBHLb.jpeg
paper_authors.fullname: zhangfan
paper_authors.type: user
paper_authors.user: Fan-s
paper_authors._id: 673ecb860ebca74a17b07271
paper_authors.name: Xiaojie Xu
paper_authors._id: 673ecb860ebca74a17b07272
paper_authors.name: Yinan He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:49:50.139Z
paper_authors._id: 62aafa49f29ff279b51f0182
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
paper_authors.fullname: yinanhe
paper_authors.type: user
paper_authors.user: ynhe
paper_authors._id: 673ecb860ebca74a17b07273
paper_authors.name: Jiashuo Yu
paper_authors._id: 673ecb860ebca74a17b07274
paper_authors.name: Ziyue Dong
paper_authors._id: 673ecb860ebca74a17b07275
paper_authors.name: Qianli Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T12:46:04.603Z
paper_authors._id: 6448b2f53e7b3c11be684348
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6448b2f53e7b3c11be684348/QvlUQG3pWf8ZyEVBV6F7w.jpeg
paper_authors.fullname: Qianli Ma
paper_authors.type: user
paper_authors.user: Mqleet
paper_authors._id: 673ecb860ebca74a17b07276
paper_authors.name: Nattapol Chanpaisit
paper_authors._id: 673ecb860ebca74a17b07277
paper_authors.name: Chenyang Si
paper_authors._id: 673ecb860ebca74a17b07278
paper_authors.name: Yuming Jiang
paper_authors._id: 673ecb860ebca74a17b07279
paper_authors.name: Yaohui Wang
paper_authors._id: 673ecb860ebca74a17b0727a
paper_authors.name: Xinyuan Chen
paper_authors._id: 673ecb860ebca74a17b0727b
paper_authors.name: Ying-Cong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:24:50.254Z
paper_authors._id: 655cba1d87b67834000590e8
paper_authors.avatarUrl: /avatars/3bd43b7c9351f65b8f38f4c8237a0146.svg
paper_authors.fullname: Yingcong Chen
paper_authors.type: user
paper_authors.user: yingcongchen
paper_authors._id: 673ecb860ebca74a17b0727c
paper_authors.name: Limin Wang
paper_authors._id: 673ecb860ebca74a17b0727d
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:24:24.368Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 673ecb860ebca74a17b0727e
paper_authors.name: Yu Qiao
paper_authors._id: 673ecb860ebca74a17b0727f
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:24:18.620Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: VBench++: Comprehensive and Versatile Benchmark Suite for Video
    Generative Models
paper_summary: Video generation has witnessed significant advancements, yet evaluating these
  models remains a challenge. A comprehensive evaluation benchmark for video
  generation is indispensable for two reasons: 1) Existing metrics do not fully
  align with human perceptions; 2) An ideal evaluation system should provide
  insights to inform future developments of video generation. To this end, we
  present VBench, a comprehensive benchmark suite that dissects "video generation
  quality" into specific, hierarchical, and disentangled dimensions, each with
  tailored prompts and evaluation methods. VBench has several appealing
  properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in
  video generation (e.g., subject identity inconsistency, motion smoothness,
  temporal flickering, and spatial relationship, etc). The evaluation metrics
  with fine-grained levels reveal individual models' strengths and weaknesses. 2)
  Human Alignment: We also provide a dataset of human preference annotations to
  validate our benchmarks' alignment with human perception, for each evaluation
  dimension respectively. 3) Valuable Insights: We look into current models'
  ability across various evaluation dimensions, and various content types. We
  also investigate the gaps between video and image generation models. 4)
  Versatile Benchmarking: VBench++ supports evaluating text-to-video and
  image-to-video. We introduce a high-quality Image Suite with an adaptive aspect
  ratio to enable fair evaluations across different image-to-video generation
  settings. Beyond assessing technical quality, VBench++ evaluates the
  trustworthiness of video generative models, providing a more holistic view of
  model performance. 5) Full Open-Sourcing: We fully open-source VBench++ and
  continually add new video generation models to our leaderboard to drive forward
  the field of video generation.
paper_ai_summary: VBench++ is a comprehensive benchmark suite for video generation that evaluates models across specific, hierarchical dimensions using fine-grained metrics and human preference annotations, offering insights into model strengths, weaknesses, and gaps compared to image generation models.
paper_ai_keywords: video generation quality subject identity inconsistency motion smoothness temporal flickering spatial relationship human preference annotations text-to-video image-to-video high-quality Image Suite adaptive aspect ratio trustworthiness video generative models