title: VideoAutoArena: An Automated Arena for Evaluating Large Multimodal
    Models in Video Analysis through User Simulation
summary: Large multimodal models (LMMs) with advanced video analysis capabilities have
  recently garnered significant attention. However, most evaluations rely on
  traditional methods like multiple-choice questions in benchmarks such as
  VideoMME and LongVideoBench, which are prone to lack the depth needed to
  capture the complex demands of real-world users. To address this limitation-and
  due to the prohibitive cost and slow pace of human annotation for video
  tasks-we introduce VideoAutoArena, an arena-style benchmark inspired by LMSYS
  Chatbot Arena's framework, designed to automatically assess LMMs' video
  analysis abilities. VideoAutoArena utilizes user simulation to generate
  open-ended, adaptive questions that rigorously assess model performance in
  video understanding. The benchmark features an automated, scalable evaluation
  framework, incorporating a modified ELO Rating System for fair and continuous
  comparisons across multiple LMMs. To validate our automated judging system, we
  construct a 'gold standard' using a carefully curated subset of human
  annotations, demonstrating that our arena strongly aligns with human judgment
  while maintaining scalability. Additionally, we introduce a fault-driven
  evolution strategy, progressively increasing question complexity to push models
  toward handling more challenging video analysis scenarios. Experimental results
  demonstrate that VideoAutoArena effectively differentiates among
  state-of-the-art LMMs, providing insights into model strengths and areas for
  improvement. To further streamline our evaluation, we introduce VideoAutoBench
  as an auxiliary benchmark, where human annotators label winners in a subset of
  VideoAutoArena battles. We use GPT-4o as a judge to compare responses against
  these human-validated answers. Together, VideoAutoArena and VideoAutoBench
  offer a cost-effective, and scalable framework for evaluating LMMs in
  user-centric video analysis.
paper_authors._id: 673ee22f63bec5e6b733a9ea
paper_authors.name: Ziyang Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T08:09:27.967Z
paper_authors._id: 6090ff099a8bcaa437b234a4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6090ff099a8bcaa437b234a4/iUvw7JXT-ngI7rGk1x-io.jpeg
paper_authors.fullname: Ziyang Luo
paper_authors.type: user
paper_authors.user: Ziyang
paper_authors._id: 673ee22f63bec5e6b733a9eb
paper_authors.name: Haoning Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T08:09:30.700Z
paper_authors._id: 63047ed2412a1b9d381b09c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63047ed2412a1b9d381b09c9/2Ill5G0uSMyGstrawgmIb.jpeg
paper_authors.fullname: Haoning Wu, Teo
paper_authors.type: user
paper_authors.user: teowu
paper_authors._id: 673ee22f63bec5e6b733a9ec
paper_authors.name: Dongxu Li
paper_authors._id: 673ee22f63bec5e6b733a9ed
paper_authors.name: Jing Ma
paper_authors._id: 673ee22f63bec5e6b733a9ee
paper_authors.name: Mohan Kankanhalli
paper_authors._id: 673ee22f63bec5e6b733a9ef
paper_authors.name: Junnan Li
paper_title: VideoAutoArena: An Automated Arena for Evaluating Large Multimodal
    Models in Video Analysis through User Simulation
paper_summary: Large multimodal models (LMMs) with advanced video analysis capabilities have
  recently garnered significant attention. However, most evaluations rely on
  traditional methods like multiple-choice questions in benchmarks such as
  VideoMME and LongVideoBench, which are prone to lack the depth needed to
  capture the complex demands of real-world users. To address this limitation-and
  due to the prohibitive cost and slow pace of human annotation for video
  tasks-we introduce VideoAutoArena, an arena-style benchmark inspired by LMSYS
  Chatbot Arena's framework, designed to automatically assess LMMs' video
  analysis abilities. VideoAutoArena utilizes user simulation to generate
  open-ended, adaptive questions that rigorously assess model performance in
  video understanding. The benchmark features an automated, scalable evaluation
  framework, incorporating a modified ELO Rating System for fair and continuous
  comparisons across multiple LMMs. To validate our automated judging system, we
  construct a 'gold standard' using a carefully curated subset of human
  annotations, demonstrating that our arena strongly aligns with human judgment
  while maintaining scalability. Additionally, we introduce a fault-driven
  evolution strategy, progressively increasing question complexity to push models
  toward handling more challenging video analysis scenarios. Experimental results
  demonstrate that VideoAutoArena effectively differentiates among
  state-of-the-art LMMs, providing insights into model strengths and areas for
  improvement. To further streamline our evaluation, we introduce VideoAutoBench
  as an auxiliary benchmark, where human annotators label winners in a subset of
  VideoAutoArena battles. We use GPT-4o as a judge to compare responses against
  these human-validated answers. Together, VideoAutoArena and VideoAutoBench
  offer a cost-effective, and scalable framework for evaluating LMMs in
  user-centric video analysis.
paper_ai_keywords: geometric mean VideoMME LongVideoBench VideoAutoArena LMSYS Chatbot Arena ELO Rating System VideoAutoBench GPT-4o VideoAutoArena VideoAutoBench