title: UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?
summary: With the rapid growth of video generative models (VGMs), it is essential to
  develop reliable and comprehensive automatic metrics for AI-generated videos
  (AIGVs). Existing methods either use off-the-shelf models optimized for other
  tasks or rely on human assessment data to train specialized evaluators. These
  approaches are constrained to specific evaluation aspects and are difficult to
  scale with the increasing demands for finer-grained and more comprehensive
  evaluations. To address this issue, this work investigates the feasibility of
  using multimodal large language models (MLLMs) as a unified evaluator for
  AIGVs, leveraging their strong visual perception and language understanding
  capabilities. To evaluate the performance of automatic metrics in unified AIGV
  evaluation, we introduce a benchmark called UVE-Bench. UVE-Bench collects
  videos generated by state-of-the-art VGMs and provides pairwise human
  preference annotations across 15 evaluation aspects. Using UVE-Bench, we
  extensively evaluate 16 MLLMs. Our empirical results suggest that while
  advanced MLLMs (e.g., Qwen2VL-72B and InternVL2.5-78B) still lag behind human
  evaluators, they demonstrate promising ability in unified AIGV evaluation,
  significantly surpassing existing specialized evaluation methods. Additionally,
  we conduct an in-depth analysis of key design choices that impact the
  performance of MLLM-driven evaluators, offering valuable insights for future
  research on AIGV evaluation. The code is available at
  https://github.com/bytedance/UVE.
paper_authors._id: 67dd70153851bd50bfa58b42
paper_authors.name: Yuanxin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-13T19:27:39.738Z
paper_authors._id: 6489761dcaea79f577897f98
paper_authors.avatarUrl: /avatars/8f56dc9c08dc2b672555602d68509a03.svg
paper_authors.fullname: Yuanxin Liu
paper_authors.type: user
paper_authors.user: lyx97
paper_authors._id: 67dd70153851bd50bfa58b43
paper_authors.name: Rui Zhu
paper_authors._id: 67dd70153851bd50bfa58b44
paper_authors.name: Shuhuai Ren
paper_authors._id: 67dd70153851bd50bfa58b45
paper_authors.name: Jiacong Wang
paper_authors._id: 67dd70153851bd50bfa58b46
paper_authors.name: Haoyuan Guo
paper_authors._id: 67dd70153851bd50bfa58b47
paper_authors.name: Xu Sun
paper_authors._id: 67dd70153851bd50bfa58b48
paper_authors.name: Lu Jiang
paper_title: UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?
paper_summary: With the rapid growth of video generative models (VGMs), it is essential to
  develop reliable and comprehensive automatic metrics for AI-generated videos
  (AIGVs). Existing methods either use off-the-shelf models optimized for other
  tasks or rely on human assessment data to train specialized evaluators. These
  approaches are constrained to specific evaluation aspects and are difficult to
  scale with the increasing demands for finer-grained and more comprehensive
  evaluations. To address this issue, this work investigates the feasibility of
  using multimodal large language models (MLLMs) as a unified evaluator for
  AIGVs, leveraging their strong visual perception and language understanding
  capabilities. To evaluate the performance of automatic metrics in unified AIGV
  evaluation, we introduce a benchmark called UVE-Bench. UVE-Bench collects
  videos generated by state-of-the-art VGMs and provides pairwise human
  preference annotations across 15 evaluation aspects. Using UVE-Bench, we
  extensively evaluate 16 MLLMs. Our empirical results suggest that while
  advanced MLLMs (e.g., Qwen2VL-72B and InternVL2.5-78B) still lag behind human
  evaluators, they demonstrate promising ability in unified AIGV evaluation,
  significantly surpassing existing specialized evaluation methods. Additionally,
  we conduct an in-depth analysis of key design choices that impact the
  performance of MLLM-driven evaluators, offering valuable insights for future
  research on AIGV evaluation. The code is available at
  https://github.com/bytedance/UVE.
paper_ai_summary: Multimodal large language models are evaluated for their effectiveness in assessing AI-generated videos across multiple aspects, demonstrating potential but still falling short of human evaluators.
paper_ai_keywords: video generative models AI-generated videos multimodal large language models UVE-Bench unified AIGV evaluation Qwen2VL-72B InternVL2.5-78B pairwise human preference annotations