UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?
With the rapid growth of video generative models (VGMs), it is essential to
develop reliable and comprehensive automatic metrics for AI-generated videos
(AIGVs). Existing methods either use off-the-shelf models optimized for other
tasks or rely on human assessment data to train specialized evaluators. These
approaches are constrained to specific evaluation aspects and are difficult to
scale with the increasing demands for finer-grained and more comprehensive
evaluations. To address this issue, this work investigates the feasibility of
using multimodal large language models (MLLMs) as a unified evaluator for
AIGVs, leveraging their strong visual perception and language understanding
capabilities. To evaluate the performance of automatic metrics in unified AIGV
evaluation, we introduce a benchmark called UVE-Bench. UVE-Bench collects
videos generated by state-of-the-art VGMs and provides pairwise human
preference annotations across 15 evaluation aspects. Using UVE-Bench, we
extensively evaluate 16 MLLMs. Our empirical results suggest that while
advanced MLLMs (e.g., Qwen2VL-72B and InternVL2.5-78B) still lag behind human
evaluators, they demonstrate promising ability in unified AIGV evaluation,
significantly surpassing existing specialized evaluation methods. Additionally,
we conduct an in-depth analysis of key design choices that impact the
performance of MLLM-driven evaluators, offering valuable insights for future
research on AIGV evaluation. The code is available at
https://github.com/bytedance/UVE.
67dd70153851bd50bfa58b42
Yuanxin Liu
claimed_verified
2025-04-13T19:27:39.738Z
6489761dcaea79f577897f98
/avatars/8f56dc9c08dc2b672555602d68509a03.svg
Yuanxin Liu
user
lyx97
67dd70153851bd50bfa58b43
Rui Zhu
67dd70153851bd50bfa58b44
Shuhuai Ren
67dd70153851bd50bfa58b45
Jiacong Wang
claimed_verified
2025-06-17T07:26:40.758Z
64d201b1c2bd235422fb1d14
/avatars/e50581aa66391cedae94e116e759b9ec.svg
wang
user
stormthunder
67dd70153851bd50bfa58b46
Haoyuan Guo
67dd70153851bd50bfa58b47
Xu Sun
67dd70153851bd50bfa58b48
Lu Jiang
UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?
With the rapid growth of video generative models (VGMs), it is essential to
develop reliable and comprehensive automatic metrics for AI-generated videos
(AIGVs). Existing methods either use off-the-shelf models optimized for other
tasks or rely on human assessment data to train specialized evaluators. These
approaches are constrained to specific evaluation aspects and are difficult to
scale with the increasing demands for finer-grained and more comprehensive
evaluations. To address this issue, this work investigates the feasibility of
using multimodal large language models (MLLMs) as a unified evaluator for
AIGVs, leveraging their strong visual perception and language understanding
capabilities. To evaluate the performance of automatic metrics in unified AIGV
evaluation, we introduce a benchmark called UVE-Bench. UVE-Bench collects
videos generated by state-of-the-art VGMs and provides pairwise human
preference annotations across 15 evaluation aspects. Using UVE-Bench, we
extensively evaluate 16 MLLMs. Our empirical results suggest that while
advanced MLLMs (e.g., Qwen2VL-72B and InternVL2.5-78B) still lag behind human
evaluators, they demonstrate promising ability in unified AIGV evaluation,
significantly surpassing existing specialized evaluation methods. Additionally,
we conduct an in-depth analysis of key design choices that impact the
performance of MLLM-driven evaluators, offering valuable insights for future
research on AIGV evaluation. The code is available at
https://github.com/bytedance/UVE.
Multimodal large language models are evaluated for their effectiveness in assessing AI-generated videos across multiple aspects, demonstrating potential but still falling short of human evaluators.
video generative models AI-generated videos multimodal large language models UVE-Bench unified AIGV evaluation Qwen2VL-72B InternVL2.5-78B pairwise human preference annotations