title: Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and
    Human Ratings
summary: While text-to-image (T2I) generative models have become ubiquitous, they do
  not necessarily generate images that align with a given prompt. While previous
  work has evaluated T2I alignment by proposing metrics, benchmarks, and
  templates for collecting human judgements, the quality of these components is
  not systematically measured. Human-rated prompt sets are generally small and
  the reliability of the ratings -- and thereby the prompt set used to compare
  models -- is not evaluated. We address this gap by performing an extensive
  study evaluating auto-eval metrics and human templates. We provide three main
  contributions: (1) We introduce a comprehensive skills-based benchmark that can
  discriminate models across different human templates. This skills-based
  benchmark categorises prompts into sub-skills, allowing a practitioner to
  pinpoint not only which skills are challenging, but at what level of complexity
  a skill becomes challenging. (2) We gather human ratings across four templates
  and four T2I models for a total of >100K annotations. This allows us to
  understand where differences arise due to inherent ambiguity in the prompt and
  where they arise due to differences in metric and model quality. (3) Finally,
  we introduce a new QA-based auto-eval metric that is better correlated with
  human ratings than existing metrics for our new dataset, across different human
  templates, and on TIFA160.
paper_authors._id: 662b2f6d82472dc87e9ec05d
paper_authors.name: Olivia Wiles
paper_authors._id: 662b2f6d82472dc87e9ec05e
paper_authors.name: Chuhan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:02:15.613Z
paper_authors._id: 6577401dab20a4deeaf40302
paper_authors.avatarUrl: /avatars/565b3a7081e30c4b603951c649b650a0.svg
paper_authors.fullname: Chuhan Zhang
paper_authors.type: user
paper_authors.user: chuhanxx
paper_authors._id: 662b2f6d82472dc87e9ec05f
paper_authors.name: Isabela Albuquerque
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:02:24.749Z
paper_authors._id: 6481bf94bae75ba5c89addd9
paper_authors.avatarUrl: /avatars/b6b2c2e828768dd6ca921a7cb30a7d6e.svg
paper_authors.fullname: Isabela Albuquerque
paper_authors.type: user
paper_authors.user: isabelaalb
paper_authors._id: 662b2f6d82472dc87e9ec060
paper_authors.name: Ivana Kajić
paper_authors._id: 662b2f6d82472dc87e9ec061
paper_authors.name: Su Wang
paper_authors._id: 662b2f6d82472dc87e9ec062
paper_authors.name: Emanuele Bugliarello
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:02:50.883Z
paper_authors._id: 626a9284783d5891f45beb53
paper_authors.avatarUrl: /avatars/8737662babc5d42cafba6087ab33e716.svg
paper_authors.fullname: Emanuele Bugliarello
paper_authors.type: user
paper_authors.user: e-bug
paper_authors._id: 662b2f6d82472dc87e9ec063
paper_authors.name: Yasumasa Onoe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:02:58.484Z
paper_authors._id: 60591867ce97b2b478a3dc4d
paper_authors.avatarUrl: /avatars/bc8fedbca5f60f99fe66b9ac37ab9a1a.svg
paper_authors.fullname: Yasumasa Onoe
paper_authors.type: user
paper_authors.user: yasumasaonoe
paper_authors._id: 662b2f6d82472dc87e9ec064
paper_authors.name: Chris Knutsen
paper_authors._id: 662b2f6d82472dc87e9ec065
paper_authors.name: Cyrus Rashtchian
paper_authors._id: 662b2f6d82472dc87e9ec066
paper_authors.name: Jordi Pont-Tuset
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-02T09:24:29.392Z
paper_authors._id: 66335b4b07f864a2f94b403d
paper_authors.avatarUrl: /avatars/0a290942c4a282b48f870500cc6424b5.svg
paper_authors.fullname: Jordi Pont Tuset
paper_authors.type: user
paper_authors.user: jponttuset
paper_authors._id: 662b2f6d82472dc87e9ec067
paper_authors.name: Aida Nematzadeh
paper_title: Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and
    Human Ratings
paper_summary: While text-to-image (T2I) generative models have become ubiquitous, they do
  not necessarily generate images that align with a given prompt. While previous
  work has evaluated T2I alignment by proposing metrics, benchmarks, and
  templates for collecting human judgements, the quality of these components is
  not systematically measured. Human-rated prompt sets are generally small and
  the reliability of the ratings -- and thereby the prompt set used to compare
  models -- is not evaluated. We address this gap by performing an extensive
  study evaluating auto-eval metrics and human templates. We provide three main
  contributions: (1) We introduce a comprehensive skills-based benchmark that can
  discriminate models across different human templates. This skills-based
  benchmark categorises prompts into sub-skills, allowing a practitioner to
  pinpoint not only which skills are challenging, but at what level of complexity
  a skill becomes challenging. (2) We gather human ratings across four templates
  and four T2I models for a total of >100K annotations. This allows us to
  understand where differences arise due to inherent ambiguity in the prompt and
  where they arise due to differences in metric and model quality. (3) Finally,
  we introduce a new QA-based auto-eval metric that is better correlated with
  human ratings than existing metrics for our new dataset, across different human
  templates, and on TIFA160.
paper_ai_summary: This study evaluates the alignment of text-to-image models with prompts by introducing a skills-based benchmark, collecting extensive human ratings, and developing a new QA-based auto-evaluation metric.
paper_ai_keywords: text-to-image auto-eval metrics human templates skills-based benchmark QA-based auto-eval metric T2I models TIFA160