Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and
  Human Ratings
While text-to-image (T2I) generative models have become ubiquitous, they do
not necessarily generate images that align with a given prompt. While previous
work has evaluated T2I alignment by proposing metrics, benchmarks, and
templates for collecting human judgements, the quality of these components is
not systematically measured. Human-rated prompt sets are generally small and
the reliability of the ratings -- and thereby the prompt set used to compare
models -- is not evaluated. We address this gap by performing an extensive
study evaluating auto-eval metrics and human templates. We provide three main
contributions: (1) We introduce a comprehensive skills-based benchmark that can
discriminate models across different human templates. This skills-based
benchmark categorises prompts into sub-skills, allowing a practitioner to
pinpoint not only which skills are challenging, but at what level of complexity
a skill becomes challenging. (2) We gather human ratings across four templates
and four T2I models for a total of >100K annotations. This allows us to
understand where differences arise due to inherent ambiguity in the prompt and
where they arise due to differences in metric and model quality. (3) Finally,
we introduce a new QA-based auto-eval metric that is better correlated with
human ratings than existing metrics for our new dataset, across different human
templates, and on TIFA160.
662b2f6d82472dc87e9ec05d
Olivia Wiles
662b2f6d82472dc87e9ec05e
Chuhan Zhang
admin_assigned
2024-04-26T12:02:15.613Z
6577401dab20a4deeaf40302
/avatars/565b3a7081e30c4b603951c649b650a0.svg
Chuhan Zhang
user
chuhanxx
662b2f6d82472dc87e9ec05f
Isabela Albuquerque
admin_assigned
2024-04-26T12:02:24.749Z
6481bf94bae75ba5c89addd9
/avatars/b6b2c2e828768dd6ca921a7cb30a7d6e.svg
Isabela Albuquerque
user
isabelaalb
662b2f6d82472dc87e9ec060
Ivana Kajić
662b2f6d82472dc87e9ec061
Su Wang
662b2f6d82472dc87e9ec062
Emanuele Bugliarello
admin_assigned
2024-04-26T12:02:50.883Z
626a9284783d5891f45beb53
/avatars/8737662babc5d42cafba6087ab33e716.svg
Emanuele Bugliarello
user
e-bug
662b2f6d82472dc87e9ec063
Yasumasa Onoe
admin_assigned
2024-04-26T12:02:58.484Z
60591867ce97b2b478a3dc4d
/avatars/bc8fedbca5f60f99fe66b9ac37ab9a1a.svg
Yasumasa Onoe
user
yasumasaonoe
662b2f6d82472dc87e9ec064
Chris Knutsen
662b2f6d82472dc87e9ec065
Cyrus Rashtchian
662b2f6d82472dc87e9ec066
Jordi Pont-Tuset
claimed_verified
2024-05-02T09:24:29.392Z
66335b4b07f864a2f94b403d
/avatars/0a290942c4a282b48f870500cc6424b5.svg
Jordi Pont Tuset
user
jponttuset
662b2f6d82472dc87e9ec067
Aida Nematzadeh
Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and
  Human Ratings
While text-to-image (T2I) generative models have become ubiquitous, they do
not necessarily generate images that align with a given prompt. While previous
work has evaluated T2I alignment by proposing metrics, benchmarks, and
templates for collecting human judgements, the quality of these components is
not systematically measured. Human-rated prompt sets are generally small and
the reliability of the ratings -- and thereby the prompt set used to compare
models -- is not evaluated. We address this gap by performing an extensive
study evaluating auto-eval metrics and human templates. We provide three main
contributions: (1) We introduce a comprehensive skills-based benchmark that can
discriminate models across different human templates. This skills-based
benchmark categorises prompts into sub-skills, allowing a practitioner to
pinpoint not only which skills are challenging, but at what level of complexity
a skill becomes challenging. (2) We gather human ratings across four templates
and four T2I models for a total of >100K annotations. This allows us to
understand where differences arise due to inherent ambiguity in the prompt and
where they arise due to differences in metric and model quality. (3) Finally,
we introduce a new QA-based auto-eval metric that is better correlated with
human ratings than existing metrics for our new dataset, across different human
templates, and on TIFA160.
This study evaluates the alignment of text-to-image models with prompts by introducing a skills-based benchmark, collecting extensive human ratings, and developing a new QA-based auto-evaluation metric.
text-to-image auto-eval metrics human templates skills-based benchmark QA-based auto-eval metric T2I models TIFA160