title: Commonsense-T2I Challenge: Can Text-to-Image Generation Models
    Understand Commonsense?
summary: We present a novel task and benchmark for evaluating the ability of
  text-to-image(T2I) generation models to produce images that fit commonsense in
  real life, which we call Commonsense-T2I. Given two adversarial text prompts
  containing an identical set of action words with minor differences, such as "a
  lightbulb without electricity" v.s. "a lightbulb with electricity", we evaluate
  whether T2I models can conduct visual-commonsense reasoning, e.g. produce
  images that fit "the lightbulb is unlit" vs. "the lightbulb is lit"
  correspondingly. Commonsense-T2I presents an adversarial challenge, providing
  pairwise text prompts along with expected outputs. The dataset is carefully
  hand-curated by experts and annotated with fine-grained labels, such as
  commonsense type and likelihood of the expected outputs, to assist analyzing
  model behavior. We benchmark a variety of state-of-the-art (sota) T2I models
  and surprisingly find that, there is still a large gap between image synthesis
  and real life photos--even the DALL-E 3 model could only achieve 48.92% on
  Commonsense-T2I, and the stable diffusion XL model only achieves 24.92%
  accuracy. Our experiments show that GPT-enriched prompts cannot solve this
  challenge, and we include a detailed analysis about possible reasons for such
  deficiency. We aim for Commonsense-T2I to serve as a high-quality evaluation
  benchmark for T2I commonsense checking, fostering advancements in real life
  image generation.
paper_authors._id: 666bb618925d891550f98440
paper_authors.name: Xingyu Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:24:05.378Z
paper_authors._id: 6336091b2db86a181ccd6054
paper_authors.avatarUrl: /avatars/4d29cef783603dc8ba123f01cde93ad8.svg
paper_authors.fullname: Xingyu Fu
paper_authors.type: user
paper_authors.user: Fiaa
paper_authors._id: 666bb618925d891550f98441
paper_authors.name: Muyu He
paper_authors._id: 666bb618925d891550f98442
paper_authors.name: Yujie Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:22:26.614Z
paper_authors._id: 633cfb6ad5935998f7582b7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633cfb6ad5935998f7582b7b/mzU4lI_GWPsYOFZHTBsYA.jpeg
paper_authors.fullname: Yujie Lu
paper_authors.type: user
paper_authors.user: FunCube
paper_authors._id: 666bb618925d891550f98443
paper_authors.name: William Yang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:24:32.258Z
paper_authors._id: 6476607efb22e3b77f3f1452
paper_authors.avatarUrl: /avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
paper_authors.fullname: William Yang Wang
paper_authors.type: user
paper_authors.user: wangwilliamyang
paper_authors._id: 666bb618925d891550f98444
paper_authors.name: Dan Roth
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:24:56.092Z
paper_authors._id: 63191aa1f18d0b5d999a47fc
paper_authors.avatarUrl: /avatars/d7eadb22aa3d8a067189b924767864e9.svg
paper_authors.fullname: Dan Roth
paper_authors.type: user
paper_authors.user: danielpaulroth
paper_title: Commonsense-T2I Challenge: Can Text-to-Image Generation Models
    Understand Commonsense?
paper_summary: We present a novel task and benchmark for evaluating the ability of
  text-to-image(T2I) generation models to produce images that fit commonsense in
  real life, which we call Commonsense-T2I. Given two adversarial text prompts
  containing an identical set of action words with minor differences, such as "a
  lightbulb without electricity" v.s. "a lightbulb with electricity", we evaluate
  whether T2I models can conduct visual-commonsense reasoning, e.g. produce
  images that fit "the lightbulb is unlit" vs. "the lightbulb is lit"
  correspondingly. Commonsense-T2I presents an adversarial challenge, providing
  pairwise text prompts along with expected outputs. The dataset is carefully
  hand-curated by experts and annotated with fine-grained labels, such as
  commonsense type and likelihood of the expected outputs, to assist analyzing
  model behavior. We benchmark a variety of state-of-the-art (sota) T2I models
  and surprisingly find that, there is still a large gap between image synthesis
  and real life photos--even the DALL-E 3 model could only achieve 48.92% on
  Commonsense-T2I, and the stable diffusion XL model only achieves 24.92%
  accuracy. Our experiments show that GPT-enriched prompts cannot solve this
  challenge, and we include a detailed analysis about possible reasons for such
  deficiency. We aim for Commonsense-T2I to serve as a high-quality evaluation
  benchmark for T2I commonsense checking, fostering advancements in real life
  image generation.
paper_ai_keywords: text-to-image (T2I) generation models visual-commonsense reasoning adversarial text prompts action words chemical-commonsense reasoning fine-grained labels state-of-the-art (sota) T2I models image synthesis real life photos DALL-E 3 stable diffusion XL GPT-enriched prompts