A Unified Agentic Framework for Evaluating Conditional Image Generation
Conditional image generation has gained significant attention for its ability
to personalize content. However, the field faces challenges in developing
task-agnostic, reliable, and explainable evaluation metrics. This paper
introduces CIGEval, a unified agentic framework for comprehensive evaluation of
conditional image generation tasks. CIGEval utilizes large multimodal models
(LMMs) as its core, integrating a multi-functional toolbox and establishing a
fine-grained evaluation framework. Additionally, we synthesize evaluation
trajectories for fine-tuning, empowering smaller LMMs to autonomously select
appropriate tools and conduct nuanced analyses based on tool outputs.
Experiments across seven prominent conditional image generation tasks
demonstrate that CIGEval (GPT-4o version) achieves a high correlation of 0.4625
with human assessments, closely matching the inter-annotator correlation of
0.47. Moreover, when implemented with 7B open-source LMMs using only 2.3K
training trajectories, CIGEval surpasses the previous GPT-4o-based
state-of-the-art method. Case studies on GPT-4o image generation highlight
CIGEval's capability in identifying subtle issues related to subject
consistency and adherence to control guidance, indicating its great potential
for automating evaluation of image generation tasks with human-level
reliability.
67f74727353d129fc7c4be7a
Jifang Wang
67f74727353d129fc7c4be7b
Xue Yang
67f74727353d129fc7c4be7c
Longyue Wang
67f74727353d129fc7c4be7d
Zhenran Xu
claimed_verified
2025-04-10T09:58:11.729Z
639c379cdb7c5f35004066cb
/avatars/3e435506ee85aa7d2d0ec2174a07462f.svg
Zhenran Xu
user
imryanxu
67f74727353d129fc7c4be7e
Yiyu Wang
67f74727353d129fc7c4be7f
Yaowei Wang
67f74727353d129fc7c4be80
Weihua Luo
67f74727353d129fc7c4be81
Kaifu Zhang
67f74727353d129fc7c4be82
Baotian Hu
67f74727353d129fc7c4be83
Min Zhang
A Unified Agentic Framework for Evaluating Conditional Image Generation
Conditional image generation has gained significant attention for its ability
to personalize content. However, the field faces challenges in developing
task-agnostic, reliable, and explainable evaluation metrics. This paper
introduces CIGEval, a unified agentic framework for comprehensive evaluation of
conditional image generation tasks. CIGEval utilizes large multimodal models
(LMMs) as its core, integrating a multi-functional toolbox and establishing a
fine-grained evaluation framework. Additionally, we synthesize evaluation
trajectories for fine-tuning, empowering smaller LMMs to autonomously select
appropriate tools and conduct nuanced analyses based on tool outputs.
Experiments across seven prominent conditional image generation tasks
demonstrate that CIGEval (GPT-4o version) achieves a high correlation of 0.4625
with human assessments, closely matching the inter-annotator correlation of
0.47. Moreover, when implemented with 7B open-source LMMs using only 2.3K
training trajectories, CIGEval surpasses the previous GPT-4o-based
state-of-the-art method. Case studies on GPT-4o image generation highlight
CIGEval's capability in identifying subtle issues related to subject
consistency and adherence to control guidance, indicating its great potential
for automating evaluation of image generation tasks with human-level
reliability.
CIGEval, a unified framework leveraging large multimodal models, provides a comprehensive and reliable evaluation mechanism for conditional image generation tasks, achieving high correlation with human assessments.
CIGEval large multimodal models multi-functional toolbox fine-grained evaluation framework evaluation trajectories fine-tuning subtle issues human-level reliability