Holistic Evaluation of Text-To-Image Models
The stunning qualitative improvement of recent text-to-image models has led
to their widespread attention and adoption. However, we lack a comprehensive
quantitative understanding of their capabilities and risks. To fill this gap,
we introduce a new benchmark, Holistic Evaluation of Text-to-Image Models
(HEIM). Whereas previous evaluations focus mostly on text-image alignment and
image quality, we identify 12 aspects, including text-image alignment, image
quality, aesthetics, originality, reasoning, knowledge, bias, toxicity,
fairness, robustness, multilinguality, and efficiency. We curate 62 scenarios
encompassing these aspects and evaluate 26 state-of-the-art text-to-image
models on this benchmark. Our results reveal that no single model excels in all
aspects, with different models demonstrating different strengths. We release
the generated images and human evaluation results for full transparency at
https://crfm.stanford.edu/heim/v1.1.0 and the code at
https://github.com/stanford-crfm/helm, which is integrated with the HELM
codebase.
654c58c7471c76068ca4e446
Tony Lee
654c58c7471c76068ca4e447
Michihiro Yasunaga
admin_assigned
2023-11-09T09:16:45.192Z
621e9388345a1d9ab65391c3
/avatars/d009c811dc63224f8eba1889fefad04b.svg
Michihiro Yasunaga
user
michiyasunaga
654c58c7471c76068ca4e448
Chenlin Meng
654c58c7471c76068ca4e449
Yifan Mai
admin_assigned
2023-11-09T09:17:08.435Z
645beb41f9d4ec91fdd935d9
/avatars/a4bddecae3096088613b915b42354b81.svg
Yifan Mai
user
yifanmai
654c58c7471c76068ca4e44a
Joon Sung Park
654c58c7471c76068ca4e44b
Agrim Gupta
admin_assigned
2023-11-09T09:17:33.515Z
63defb9015266dd945f9f772
/avatars/b304ac6c3b45553358f8b6cd5c3df293.svg
Agrim
user
agrimgupta
654c58c7471c76068ca4e44c
Yunzhi Zhang
admin_assigned
2023-11-09T09:17:45.822Z
63793aee2cb50dda39d6e73b
/avatars/1d7eb1a19ce2e18802e9481200eb03f7.svg
Yunzhi Zhang
user
Yunzhi
654c58c7471c76068ca4e44d
Deepak Narayanan
654c58c7471c76068ca4e44e
Hannah Benita Teufel
654c58c7471c76068ca4e44f
Marco Bellagente
admin_assigned
2023-11-09T09:18:44.191Z
63519eb7a8e595171ab179cb
/avatars/792789acb90450962c9cdbec5b24e963.svg
Marco Bellagente
user
marcob
654c58c7471c76068ca4e450
Minguk Kang
654c58c7471c76068ca4e451
Taesung Park
admin_assigned
2023-11-09T09:19:18.987Z
637563c59ef4e3b87a60d503
/avatars/8264de115e6305e4720c78fc7dce7f8d.svg
Taesung Park
user
taesungp
654c58c7471c76068ca4e452
Jure Leskovec
654c58c7471c76068ca4e453
Jun-Yan Zhu
admin_assigned
2023-11-09T09:19:31.308Z
63a0acc32fabbbb899952a2b
https://cdn-avatars.huggingface.co/v1/production/uploads/1671474335794-noauth.jpeg
Jun-Yan Zhu
user
junyanz
654c58c7471c76068ca4e454
Li Fei-Fei
654c58c7471c76068ca4e455
Jiajun Wu
admin_assigned
2023-11-09T09:19:56.314Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
654c58c7471c76068ca4e456
Stefano Ermon
admin_assigned
2023-11-09T09:20:04.971Z
62f6e244329d4d014d1f4ac5
/avatars/5a8b2bb063c2ebc340504b22530f6811.svg
Stefano Ermon
user
ermonste
654c58c7471c76068ca4e457
Percy Liang
admin_assigned
2023-11-09T09:20:13.067Z
6409651b9e9f790c905b2335
/avatars/1fb8c80b60f21f65a0a027319101f236.svg
Percy Liang
user
percyliang
Holistic Evaluation of Text-To-Image Models
The stunning qualitative improvement of recent text-to-image models has led
to their widespread attention and adoption. However, we lack a comprehensive
quantitative understanding of their capabilities and risks. To fill this gap,
we introduce a new benchmark, Holistic Evaluation of Text-to-Image Models
(HEIM). Whereas previous evaluations focus mostly on text-image alignment and
image quality, we identify 12 aspects, including text-image alignment, image
quality, aesthetics, originality, reasoning, knowledge, bias, toxicity,
fairness, robustness, multilinguality, and efficiency. We curate 62 scenarios
encompassing these aspects and evaluate 26 state-of-the-art text-to-image
models on this benchmark. Our results reveal that no single model excels in all
aspects, with different models demonstrating different strengths. We release
the generated images and human evaluation results for full transparency at
https://crfm.stanford.edu/heim/v1.1.0 and the code at
https://github.com/stanford-crfm/helm, which is integrated with the HELM
codebase.
A new benchmark evaluates text-to-image models across 12 aspects, revealing no single model excels comprehensively in all areas.
text-to-image models Holistic Evaluation of Text-to-Image Models HEIM text-image alignment image quality aesthetics originality reasoning knowledge bias toxicity fairness robustness multilinguality efficiency