title: A Unified Agentic Framework for Evaluating Conditional Image Generation
summary: Conditional image generation has gained significant attention for its ability
  to personalize content. However, the field faces challenges in developing
  task-agnostic, reliable, and explainable evaluation metrics. This paper
  introduces CIGEval, a unified agentic framework for comprehensive evaluation of
  conditional image generation tasks. CIGEval utilizes large multimodal models
  (LMMs) as its core, integrating a multi-functional toolbox and establishing a
  fine-grained evaluation framework. Additionally, we synthesize evaluation
  trajectories for fine-tuning, empowering smaller LMMs to autonomously select
  appropriate tools and conduct nuanced analyses based on tool outputs.
  Experiments across seven prominent conditional image generation tasks
  demonstrate that CIGEval (GPT-4o version) achieves a high correlation of 0.4625
  with human assessments, closely matching the inter-annotator correlation of
  0.47. Moreover, when implemented with 7B open-source LMMs using only 2.3K
  training trajectories, CIGEval surpasses the previous GPT-4o-based
  state-of-the-art method. Case studies on GPT-4o image generation highlight
  CIGEval's capability in identifying subtle issues related to subject
  consistency and adherence to control guidance, indicating its great potential
  for automating evaluation of image generation tasks with human-level
  reliability.
paper_authors._id: 67f74727353d129fc7c4be7a
paper_authors.name: Jifang Wang
paper_authors._id: 67f74727353d129fc7c4be7b
paper_authors.name: Xue Yang
paper_authors._id: 67f74727353d129fc7c4be7c
paper_authors.name: Longyue Wang
paper_authors._id: 67f74727353d129fc7c4be7d
paper_authors.name: Zhenran Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T09:58:11.729Z
paper_authors._id: 639c379cdb7c5f35004066cb
paper_authors.avatarUrl: /avatars/3e435506ee85aa7d2d0ec2174a07462f.svg
paper_authors.fullname: Zhenran Xu
paper_authors.type: user
paper_authors.user: imryanxu
paper_authors._id: 67f74727353d129fc7c4be7e
paper_authors.name: Yiyu Wang
paper_authors._id: 67f74727353d129fc7c4be7f
paper_authors.name: Yaowei Wang
paper_authors._id: 67f74727353d129fc7c4be80
paper_authors.name: Weihua Luo
paper_authors._id: 67f74727353d129fc7c4be81
paper_authors.name: Kaifu Zhang
paper_authors._id: 67f74727353d129fc7c4be82
paper_authors.name: Baotian Hu
paper_authors._id: 67f74727353d129fc7c4be83
paper_authors.name: Min Zhang
paper_title: A Unified Agentic Framework for Evaluating Conditional Image Generation
paper_summary: Conditional image generation has gained significant attention for its ability
  to personalize content. However, the field faces challenges in developing
  task-agnostic, reliable, and explainable evaluation metrics. This paper
  introduces CIGEval, a unified agentic framework for comprehensive evaluation of
  conditional image generation tasks. CIGEval utilizes large multimodal models
  (LMMs) as its core, integrating a multi-functional toolbox and establishing a
  fine-grained evaluation framework. Additionally, we synthesize evaluation
  trajectories for fine-tuning, empowering smaller LMMs to autonomously select
  appropriate tools and conduct nuanced analyses based on tool outputs.
  Experiments across seven prominent conditional image generation tasks
  demonstrate that CIGEval (GPT-4o version) achieves a high correlation of 0.4625
  with human assessments, closely matching the inter-annotator correlation of
  0.47. Moreover, when implemented with 7B open-source LMMs using only 2.3K
  training trajectories, CIGEval surpasses the previous GPT-4o-based
  state-of-the-art method. Case studies on GPT-4o image generation highlight
  CIGEval's capability in identifying subtle issues related to subject
  consistency and adherence to control guidance, indicating its great potential
  for automating evaluation of image generation tasks with human-level
  reliability.
paper_ai_summary: CIGEval, a unified framework leveraging large multimodal models, provides a comprehensive and reliable evaluation mechanism for conditional image generation tasks, achieving high correlation with human assessments.
paper_ai_keywords: CIGEval large multimodal models multi-functional toolbox fine-grained evaluation framework evaluation trajectories fine-tuning subtle issues human-level reliability