title: VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC
    Videos
summary: MLLMs have been widely studied for video question answering recently.
  However, most existing assessments focus on natural videos, overlooking
  synthetic videos, such as AI-generated content (AIGC). Meanwhile, some works in
  video generation rely on MLLMs to evaluate the quality of generated videos, but
  the capabilities of MLLMs on interpreting AIGC videos remain largely
  underexplored. To address this, we propose a new benchmark, VF-Eval, which
  introduces four tasks-coherence validation, error awareness, error type
  detection, and reasoning evaluation-to comprehensively evaluate the abilities
  of MLLMs on AIGC videos. We evaluate 13 frontier MLLMs on VF-Eval and find that
  even the best-performing model, GPT-4.1, struggles to achieve consistently good
  performance across all tasks. This highlights the challenging nature of our
  benchmark. Additionally, to investigate the practical applications of VF-Eval
  in improving video generation, we conduct an experiment, RePrompt,
  demonstrating that aligning MLLMs more closely with human feedback can benefit
  video generation.
paper_authors._id: 68390e95b85141ce6c11b50f
paper_authors.name: Tingyu Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:12.429Z
paper_authors._id: 64dc29d9b5d625e0e9a6ecb9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QxGBsnk1cNsBEPqSx4ae-.jpeg
paper_authors.fullname: Tingyu Song
paper_authors.type: user
paper_authors.user: songtingyu
paper_authors._id: 68390e95b85141ce6c11b510
paper_authors.name: Tongyan Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:15.154Z
paper_authors._id: 66e83ec5deb449d8d856e78d
paper_authors.avatarUrl: /avatars/c5e56be65fcacb3192ce10ba6d8f48e2.svg
paper_authors.fullname: Tongyan Hu
paper_authors.type: user
paper_authors.user: entropyhu
paper_authors._id: 68390e95b85141ce6c11b511
paper_authors.name: Guo Gan
paper_authors._id: 68390e95b85141ce6c11b512
paper_authors.name: Yilun Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:47:26.006Z
paper_authors._id: 62f662bcc58915315c4eccea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
paper_authors.fullname: Yilun Zhao
paper_authors.type: user
paper_authors.user: yilunzhao
paper_title: VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC
    Videos
paper_summary: MLLMs have been widely studied for video question answering recently.
  However, most existing assessments focus on natural videos, overlooking
  synthetic videos, such as AI-generated content (AIGC). Meanwhile, some works in
  video generation rely on MLLMs to evaluate the quality of generated videos, but
  the capabilities of MLLMs on interpreting AIGC videos remain largely
  underexplored. To address this, we propose a new benchmark, VF-Eval, which
  introduces four tasks-coherence validation, error awareness, error type
  detection, and reasoning evaluation-to comprehensively evaluate the abilities
  of MLLMs on AIGC videos. We evaluate 13 frontier MLLMs on VF-Eval and find that
  even the best-performing model, GPT-4.1, struggles to achieve consistently good
  performance across all tasks. This highlights the challenging nature of our
  benchmark. Additionally, to investigate the practical applications of VF-Eval
  in improving video generation, we conduct an experiment, RePrompt,
  demonstrating that aligning MLLMs more closely with human feedback can benefit
  video generation.
paper_ai_summary: A new benchmark, VF-Eval, evaluates the capabilities of MLLMs in interpreting AI-generated content videos across four tasks, highlighting challenges and demonstrating benefits in video generation through human feedback alignment.
paper_ai_keywords: MLLMs video question answering synthetic videos AI-generated content (AIGC) VF-Eval coherence validation error awareness error type detection reasoning evaluation GPT-4.1 RePrompt