Judge Anything: MLLM as a Judge Across Any Modality
Evaluating generative foundation models on open-ended multimodal
understanding (MMU) and generation (MMG) tasks across diverse modalities (e.g.,
images, audio, video) poses significant challenges due to the complexity of
cross-modal interactions. To this end, the idea of utilizing Multimodal LLMs
(MLLMs) as automated judges has emerged, with encouraging results in assessing
vision-language understanding tasks. Moving further, this paper extends
MLLM-as-a-Judge across modalities to a unified manner by introducing two
benchmarks, TaskAnything and JudgeAnything, to respectively evaluate the
overall performance and judging capabilities of MLLMs across any-to-any
modality tasks. Specifically, TaskAnything evaluates the MMU and MMG
capabilities across 15 any-to-any modality categories, employing 1,500 queries
curated from well-established benchmarks. Furthermore, JudgeAnything evaluates
the judging capabilities of 5 advanced (e.g., GPT-4o and Gemini-2.0-Flash) from
the perspectives of Pair Comparison and Score Evaluation, providing a
standardized testbed that incorporates human judgments and detailed rubrics.
Our extensive experiments reveal that while these MLLMs show promise in
assessing MMU (i.e., achieving an average of 66.55% in Pair Comparison setting
and 42.79% in Score Evaluation setting), they encounter significant challenges
with MMG tasks (i.e., averaging only 53.37% in Pair Comparison setting and
30.05% in Score Evaluation setting), exposing cross-modality biases and
hallucination issues. To address this, we present OmniArena, an automated
platform for evaluating omni-models and multimodal reward models. Our work
highlights the need for fairer evaluation protocols and stronger alignment with
human preferences. The source code and dataset are publicly available at:
https://urrealhero.github.io/judgeanythingweb/.
67e21f300e6b6fcc3eb38ae1
Shu Pu
claimed_verified
2025-03-26T20:49:18.479Z
663ba53b96c98cfb8ded1c4f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/6-ipDyZbMYj_lfVurLjvh.png
Shu Pu
user
pudashi
67e21f300e6b6fcc3eb38ae2
Yaochen Wang
67e21f300e6b6fcc3eb38ae3
Dongping Chen
admin_assigned
2025-03-25T09:20:55.986Z
65e2be1e630e2db23829ee8d
/avatars/294f9ba909037f03669dc0bb80cabfe3.svg
Dongping Chen
user
fjchendp
67e21f300e6b6fcc3eb38ae4
Yuhang Chen
admin_assigned
2025-03-25T09:21:02.911Z
64964aae457f60023c6a6f9d
/avatars/342603e0028204f33fe7f5e3f3da1aa3.svg
Yuhang Chen
user
yuhangchen
67e21f300e6b6fcc3eb38ae5
Guohao Wang
admin_assigned
2025-03-25T09:21:09.345Z
67c94fd48670a35a7c05f36c
/avatars/a59a7872bcc58fec7747225f2d3da3f9.svg
Guohao Wang
user
NiuniuWang
67e21f300e6b6fcc3eb38ae6
Qi Qin
67e21f300e6b6fcc3eb38ae7
Zhongyi Zhang
67e21f300e6b6fcc3eb38ae8
Zhiyuan Zhang
67e21f300e6b6fcc3eb38ae9
Zetong Zhou
admin_assigned
2025-03-25T09:21:29.118Z
6697e7e55ef2828a1ff371c3
/avatars/b361ea817760f7cb5c5d39028ee6b507.svg
Zetong Zhou
user
Frywind
67e21f300e6b6fcc3eb38aea
Shuang Gong
admin_assigned
2025-03-25T09:21:35.465Z
67575cac2f7acf9a8b4626fa
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1OkoZh8A4jPKHpTg5iSXP.png
Shuang Gong
user
shuang72
67e21f300e6b6fcc3eb38aeb
Yi Gui
67e21f300e6b6fcc3eb38aec
Yao Wan
67e21f300e6b6fcc3eb38aed
Philip S. Yu
Judge Anything: MLLM as a Judge Across Any Modality
Evaluating generative foundation models on open-ended multimodal
understanding (MMU) and generation (MMG) tasks across diverse modalities (e.g.,
images, audio, video) poses significant challenges due to the complexity of
cross-modal interactions. To this end, the idea of utilizing Multimodal LLMs
(MLLMs) as automated judges has emerged, with encouraging results in assessing
vision-language understanding tasks. Moving further, this paper extends
MLLM-as-a-Judge across modalities to a unified manner by introducing two
benchmarks, TaskAnything and JudgeAnything, to respectively evaluate the
overall performance and judging capabilities of MLLMs across any-to-any
modality tasks. Specifically, TaskAnything evaluates the MMU and MMG
capabilities across 15 any-to-any modality categories, employing 1,500 queries
curated from well-established benchmarks. Furthermore, JudgeAnything evaluates
the judging capabilities of 5 advanced (e.g., GPT-4o and Gemini-2.0-Flash) from
the perspectives of Pair Comparison and Score Evaluation, providing a
standardized testbed that incorporates human judgments and detailed rubrics.
Our extensive experiments reveal that while these MLLMs show promise in
assessing MMU (i.e., achieving an average of 66.55% in Pair Comparison setting
and 42.79% in Score Evaluation setting), they encounter significant challenges
with MMG tasks (i.e., averaging only 53.37% in Pair Comparison setting and
30.05% in Score Evaluation setting), exposing cross-modality biases and
hallucination issues. To address this, we present OmniArena, an automated
platform for evaluating omni-models and multimodal reward models. Our work
highlights the need for fairer evaluation protocols and stronger alignment with
human preferences. The source code and dataset are publicly available at:
https://urrealhero.github.io/judgeanythingweb/.
MLLMs evaluate cross-modal understanding and generation tasks across various modalities but face challenges with MMG tasks, leading to the proposal of OmniArena for more comprehensive evaluation.
Multimodal LLMs MLLM-as-a-Judge TaskAnything JudgeAnything MMU MMG Pair Comparison Score Evaluation OmniArena cross-modality biases hallucination issues