title: Redundancy Principles for MLLMs Benchmarks
summary: With the rapid iteration of Multi-modality Large Language Models (MLLMs) and
  the evolving demands of the field, the number of benchmarks produced annually
  has surged into the hundreds. The rapid growth has inevitably led to
  significant redundancy among benchmarks. Therefore, it is crucial to take a
  step back and critically assess the current state of redundancy and propose
  targeted principles for constructing effective MLLM benchmarks. In this paper,
  we focus on redundancy from three key perspectives: 1) Redundancy of benchmark
  capability dimensions, 2) Redundancy in the number of test questions, and 3)
  Cross-benchmark redundancy within specific domains. Through the comprehensive
  analysis over hundreds of MLLMs' performance across more than 20 benchmarks, we
  aim to quantitatively measure the level of redundancy lies in existing MLLM
  evaluations, provide valuable insights to guide the future development of MLLM
  benchmarks, and offer strategies to refine and address redundancy issues
  effectively.
paper_authors._id: 67973e05495916be7c0086cc
paper_authors.name: Zicheng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T14:56:43.367Z
paper_authors._id: 6526cc6bab4f5d98382f5603
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6526cc6bab4f5d98382f5603/NekYYb61I5nt4au6gXsVK.jpeg
paper_authors.fullname: Zicheng Zhang
paper_authors.type: user
paper_authors.user: zhangzicheng
paper_authors._id: 67973e05495916be7c0086cd
paper_authors.name: Xiangyu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:48:19.846Z
paper_authors._id: 6530e62f536dbca918e71c3e
paper_authors.avatarUrl: /avatars/efc93bc767e561c6c6d429f65c23382d.svg
paper_authors.fullname: Xiangyu Z
paper_authors.type: user
paper_authors.user: PhoenixZ
paper_authors._id: 67973e05495916be7c0086ce
paper_authors.name: Xinyu Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-27T10:43:11.727Z
paper_authors._id: 64f5f8dd9b17cd59c453c57f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
paper_authors.fullname: Xinyu Fang
paper_authors.type: user
paper_authors.user: nebulae09
paper_authors._id: 67973e05495916be7c0086cf
paper_authors.name: Chunyi Li
paper_authors._id: 67973e05495916be7c0086d0
paper_authors.name: Xiaohong Liu
paper_authors._id: 67973e05495916be7c0086d1
paper_authors.name: Xiongkuo Min
paper_authors._id: 67973e05495916be7c0086d2
paper_authors.name: Haodong Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-27T10:43:09.483Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 67973e05495916be7c0086d3
paper_authors.name: Kai Chen
paper_authors._id: 67973e05495916be7c0086d4
paper_authors.name: Guangtao Zhai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T14:58:51.908Z
paper_authors._id: 65535b125413c1a54e6fb243
paper_authors.avatarUrl: /avatars/03bcf1d58865f5406aff49a415e78bdc.svg
paper_authors.fullname: Guangtao Zhai
paper_authors.type: user
paper_authors.user: GTZhai
paper_title: Redundancy Principles for MLLMs Benchmarks
paper_summary: With the rapid iteration of Multi-modality Large Language Models (MLLMs) and
  the evolving demands of the field, the number of benchmarks produced annually
  has surged into the hundreds. The rapid growth has inevitably led to
  significant redundancy among benchmarks. Therefore, it is crucial to take a
  step back and critically assess the current state of redundancy and propose
  targeted principles for constructing effective MLLM benchmarks. In this paper,
  we focus on redundancy from three key perspectives: 1) Redundancy of benchmark
  capability dimensions, 2) Redundancy in the number of test questions, and 3)
  Cross-benchmark redundancy within specific domains. Through the comprehensive
  analysis over hundreds of MLLMs' performance across more than 20 benchmarks, we
  aim to quantitatively measure the level of redundancy lies in existing MLLM
  evaluations, provide valuable insights to guide the future development of MLLM
  benchmarks, and offer strategies to refine and address redundancy issues
  effectively.
paper_ai_keywords: sequence-to-sequence