title: Judge Anything: MLLM as a Judge Across Any Modality
summary: Evaluating generative foundation models on open-ended multimodal
  understanding (MMU) and generation (MMG) tasks across diverse modalities (e.g.,
  images, audio, video) poses significant challenges due to the complexity of
  cross-modal interactions. To this end, the idea of utilizing Multimodal LLMs
  (MLLMs) as automated judges has emerged, with encouraging results in assessing
  vision-language understanding tasks. Moving further, this paper extends
  MLLM-as-a-Judge across modalities to a unified manner by introducing two
  benchmarks, TaskAnything and JudgeAnything, to respectively evaluate the
  overall performance and judging capabilities of MLLMs across any-to-any
  modality tasks. Specifically, TaskAnything evaluates the MMU and MMG
  capabilities across 15 any-to-any modality categories, employing 1,500 queries
  curated from well-established benchmarks. Furthermore, JudgeAnything evaluates
  the judging capabilities of 5 advanced (e.g., GPT-4o and Gemini-2.0-Flash) from
  the perspectives of Pair Comparison and Score Evaluation, providing a
  standardized testbed that incorporates human judgments and detailed rubrics.
  Our extensive experiments reveal that while these MLLMs show promise in
  assessing MMU (i.e., achieving an average of 66.55% in Pair Comparison setting
  and 42.79% in Score Evaluation setting), they encounter significant challenges
  with MMG tasks (i.e., averaging only 53.37% in Pair Comparison setting and
  30.05% in Score Evaluation setting), exposing cross-modality biases and
  hallucination issues. To address this, we present OmniArena, an automated
  platform for evaluating omni-models and multimodal reward models. Our work
  highlights the need for fairer evaluation protocols and stronger alignment with
  human preferences. The source code and dataset are publicly available at:
  https://urrealhero.github.io/judgeanythingweb/.
paper_authors._id: 67e21f300e6b6fcc3eb38ae1
paper_authors.name: Shu Pu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:49:18.479Z
paper_authors._id: 663ba53b96c98cfb8ded1c4f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/6-ipDyZbMYj_lfVurLjvh.png
paper_authors.fullname: Shu Pu
paper_authors.type: user
paper_authors.user: pudashi
paper_authors._id: 67e21f300e6b6fcc3eb38ae2
paper_authors.name: Yaochen Wang
paper_authors._id: 67e21f300e6b6fcc3eb38ae3
paper_authors.name: Dongping Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:20:55.986Z
paper_authors._id: 65e2be1e630e2db23829ee8d
paper_authors.avatarUrl: /avatars/294f9ba909037f03669dc0bb80cabfe3.svg
paper_authors.fullname: Dongping Chen
paper_authors.type: user
paper_authors.user: fjchendp
paper_authors._id: 67e21f300e6b6fcc3eb38ae4
paper_authors.name: Yuhang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:21:02.911Z
paper_authors._id: 64964aae457f60023c6a6f9d
paper_authors.avatarUrl: /avatars/342603e0028204f33fe7f5e3f3da1aa3.svg
paper_authors.fullname: Yuhang Chen
paper_authors.type: user
paper_authors.user: yuhangchen
paper_authors._id: 67e21f300e6b6fcc3eb38ae5
paper_authors.name: Guohao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:21:09.345Z
paper_authors._id: 67c94fd48670a35a7c05f36c
paper_authors.avatarUrl: /avatars/a59a7872bcc58fec7747225f2d3da3f9.svg
paper_authors.fullname: Guohao Wang
paper_authors.type: user
paper_authors.user: NiuniuWang
paper_authors._id: 67e21f300e6b6fcc3eb38ae6
paper_authors.name: Qi Qin
paper_authors._id: 67e21f300e6b6fcc3eb38ae7
paper_authors.name: Zhongyi Zhang
paper_authors._id: 67e21f300e6b6fcc3eb38ae8
paper_authors.name: Zhiyuan Zhang
paper_authors._id: 67e21f300e6b6fcc3eb38ae9
paper_authors.name: Zetong Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:21:29.118Z
paper_authors._id: 6697e7e55ef2828a1ff371c3
paper_authors.avatarUrl: /avatars/b361ea817760f7cb5c5d39028ee6b507.svg
paper_authors.fullname: Zetong Zhou
paper_authors.type: user
paper_authors.user: Frywind
paper_authors._id: 67e21f300e6b6fcc3eb38aea
paper_authors.name: Shuang Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:21:35.465Z
paper_authors._id: 67575cac2f7acf9a8b4626fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1OkoZh8A4jPKHpTg5iSXP.png
paper_authors.fullname: Shuang Gong
paper_authors.type: user
paper_authors.user: shuang72
paper_authors._id: 67e21f300e6b6fcc3eb38aeb
paper_authors.name: Yi Gui
paper_authors._id: 67e21f300e6b6fcc3eb38aec
paper_authors.name: Yao Wan
paper_authors._id: 67e21f300e6b6fcc3eb38aed
paper_authors.name: Philip S. Yu
paper_title: Judge Anything: MLLM as a Judge Across Any Modality
paper_summary: Evaluating generative foundation models on open-ended multimodal
  understanding (MMU) and generation (MMG) tasks across diverse modalities (e.g.,
  images, audio, video) poses significant challenges due to the complexity of
  cross-modal interactions. To this end, the idea of utilizing Multimodal LLMs
  (MLLMs) as automated judges has emerged, with encouraging results in assessing
  vision-language understanding tasks. Moving further, this paper extends
  MLLM-as-a-Judge across modalities to a unified manner by introducing two
  benchmarks, TaskAnything and JudgeAnything, to respectively evaluate the
  overall performance and judging capabilities of MLLMs across any-to-any
  modality tasks. Specifically, TaskAnything evaluates the MMU and MMG
  capabilities across 15 any-to-any modality categories, employing 1,500 queries
  curated from well-established benchmarks. Furthermore, JudgeAnything evaluates
  the judging capabilities of 5 advanced (e.g., GPT-4o and Gemini-2.0-Flash) from
  the perspectives of Pair Comparison and Score Evaluation, providing a
  standardized testbed that incorporates human judgments and detailed rubrics.
  Our extensive experiments reveal that while these MLLMs show promise in
  assessing MMU (i.e., achieving an average of 66.55% in Pair Comparison setting
  and 42.79% in Score Evaluation setting), they encounter significant challenges
  with MMG tasks (i.e., averaging only 53.37% in Pair Comparison setting and
  30.05% in Score Evaluation setting), exposing cross-modality biases and
  hallucination issues. To address this, we present OmniArena, an automated
  platform for evaluating omni-models and multimodal reward models. Our work
  highlights the need for fairer evaluation protocols and stronger alignment with
  human preferences. The source code and dataset are publicly available at:
  https://urrealhero.github.io/judgeanythingweb/.
paper_ai_summary: MLLMs evaluate cross-modal understanding and generation tasks across various modalities but face challenges with MMG tasks, leading to the proposal of OmniArena for more comprehensive evaluation.
paper_ai_keywords: Multimodal LLMs MLLM-as-a-Judge TaskAnything JudgeAnything MMU MMG Pair Comparison Score Evaluation OmniArena cross-modality biases hallucination issues