title: ImagenHub: Standardizing the evaluation of conditional image generation
    models
summary: Recently, a myriad of conditional image generation and editing models have
  been developed to serve different downstream tasks, including text-to-image
  generation, text-guided image editing, subject-driven image generation,
  control-guided image generation, etc. However, we observe huge inconsistencies
  in experimental conditions: datasets, inference, and evaluation metrics -
  render fair comparisons difficult. This paper proposes ImagenHub, which is a
  one-stop library to standardize the inference and evaluation of all the
  conditional image generation models. Firstly, we define seven prominent tasks
  and curate high-quality evaluation datasets for them. Secondly, we built a
  unified inference pipeline to ensure fair comparison. Thirdly, we design two
  human evaluation scores, i.e. Semantic Consistency and Perceptual Quality,
  along with comprehensive guidelines to evaluate generated images. We train
  expert raters to evaluate the model outputs based on the proposed metrics. Our
  human evaluation achieves a high inter-worker agreement of Krippendorff's alpha
  on 76% models with a value higher than 0.4. We comprehensively evaluated a
  total of around 30 models and observed three key takeaways: (1) the existing
  models' performance is generally unsatisfying except for Text-guided Image
  Generation and Subject-driven Image Generation, with 74% models achieving an
  overall score lower than 0.5. (2) we examined the claims from published papers
  and found 83% of them hold with a few exceptions. (3) None of the existing
  automatic metrics has a Spearman's correlation higher than 0.2 except
  subject-driven image generation. Moving forward, we will continue our efforts
  to evaluate newly published models and update our leaderboard to keep track of
  the progress in conditional image generation.
paper_authors._id: 651cdb74312b6fd1c6eb6eec
paper_authors.name: Max Ku
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-05T06:34:04.408Z
paper_authors._id: 631d760344503b7227837242
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631d760344503b7227837242/3b6JRusFX6GKJpsN9ZdeJ.png
paper_authors.fullname: Max Ku
paper_authors.type: user
paper_authors.user: vinesmsuic
paper_authors._id: 651cdb74312b6fd1c6eb6eed
paper_authors.name: Tianle Li
paper_authors._id: 651cdb74312b6fd1c6eb6eee
paper_authors.name: Kai Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-30T09:30:37.834Z
paper_authors._id: 63e0a50242591dda0b9dca5c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
paper_authors.fullname: Kai Zhang
paper_authors.type: user
paper_authors.user: drogozhang
paper_authors._id: 651cdb74312b6fd1c6eb6eef
paper_authors.name: Yujie Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-05T07:31:06.369Z
paper_authors._id: 633cfb6ad5935998f7582b7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633cfb6ad5935998f7582b7b/mzU4lI_GWPsYOFZHTBsYA.jpeg
paper_authors.fullname: Yujie Lu
paper_authors.type: user
paper_authors.user: FunCube
paper_authors._id: 651cdb74312b6fd1c6eb6ef0
paper_authors.name: Xingyu Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-19T07:39:45.830Z
paper_authors._id: 6336091b2db86a181ccd6054
paper_authors.avatarUrl: /avatars/4d29cef783603dc8ba123f01cde93ad8.svg
paper_authors.fullname: Xingyu Fu
paper_authors.type: user
paper_authors.user: Fiaa
paper_authors._id: 651cdb74312b6fd1c6eb6ef1
paper_authors.name: Wenwen Zhuang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-11T07:35:08.258Z
paper_authors._id: 642f83089b2484d7d8568787
paper_authors.avatarUrl: /avatars/5c502332d23d7332b23fc45c704782ac.svg
paper_authors.fullname: Wenwen Zhuang
paper_authors.type: user
paper_authors.user: Anwen0809
paper_authors._id: 651cdb74312b6fd1c6eb6ef2
paper_authors.name: Wenhu Chen
paper_title: ImagenHub: Standardizing the evaluation of conditional image generation
    models
paper_summary: Recently, a myriad of conditional image generation and editing models have
  been developed to serve different downstream tasks, including text-to-image
  generation, text-guided image editing, subject-driven image generation,
  control-guided image generation, etc. However, we observe huge inconsistencies
  in experimental conditions: datasets, inference, and evaluation metrics -
  render fair comparisons difficult. This paper proposes ImagenHub, which is a
  one-stop library to standardize the inference and evaluation of all the
  conditional image generation models. Firstly, we define seven prominent tasks
  and curate high-quality evaluation datasets for them. Secondly, we built a
  unified inference pipeline to ensure fair comparison. Thirdly, we design two
  human evaluation scores, i.e. Semantic Consistency and Perceptual Quality,
  along with comprehensive guidelines to evaluate generated images. We train
  expert raters to evaluate the model outputs based on the proposed metrics. Our
  human evaluation achieves a high inter-worker agreement of Krippendorff's alpha
  on 76% models with a value higher than 0.4. We comprehensively evaluated a
  total of around 30 models and observed three key takeaways: (1) the existing
  models' performance is generally unsatisfying except for Text-guided Image
  Generation and Subject-driven Image Generation, with 74% models achieving an
  overall score lower than 0.5. (2) we examined the claims from published papers
  and found 83% of them hold with a few exceptions. (3) None of the existing
  automatic metrics has a Spearman's correlation higher than 0.2 except
  subject-driven image generation. Moving forward, we will continue our efforts
  to evaluate newly published models and update our leaderboard to keep track of
  the progress in conditional image generation.