title: Holistic Evaluation of Text-To-Image Models
summary: The stunning qualitative improvement of recent text-to-image models has led
  to their widespread attention and adoption. However, we lack a comprehensive
  quantitative understanding of their capabilities and risks. To fill this gap,
  we introduce a new benchmark, Holistic Evaluation of Text-to-Image Models
  (HEIM). Whereas previous evaluations focus mostly on text-image alignment and
  image quality, we identify 12 aspects, including text-image alignment, image
  quality, aesthetics, originality, reasoning, knowledge, bias, toxicity,
  fairness, robustness, multilinguality, and efficiency. We curate 62 scenarios
  encompassing these aspects and evaluate 26 state-of-the-art text-to-image
  models on this benchmark. Our results reveal that no single model excels in all
  aspects, with different models demonstrating different strengths. We release
  the generated images and human evaluation results for full transparency at
  https://crfm.stanford.edu/heim/v1.1.0 and the code at
  https://github.com/stanford-crfm/helm, which is integrated with the HELM
  codebase.
paper_authors._id: 654c58c7471c76068ca4e446
paper_authors.name: Tony Lee
paper_authors._id: 654c58c7471c76068ca4e447
paper_authors.name: Michihiro Yasunaga
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:16:45.192Z
paper_authors._id: 621e9388345a1d9ab65391c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621e9388345a1d9ab65391c3/RxurNzyAWJOUdgeSHQi1R.jpeg
paper_authors.fullname: Michihiro Yasunaga
paper_authors.type: user
paper_authors.user: michiyasunaga
paper_authors._id: 654c58c7471c76068ca4e448
paper_authors.name: Chenlin Meng
paper_authors._id: 654c58c7471c76068ca4e449
paper_authors.name: Yifan Mai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:17:08.435Z
paper_authors._id: 645beb41f9d4ec91fdd935d9
paper_authors.avatarUrl: /avatars/a4bddecae3096088613b915b42354b81.svg
paper_authors.fullname: Yifan Mai
paper_authors.type: user
paper_authors.user: yifanmai
paper_authors._id: 654c58c7471c76068ca4e44a
paper_authors.name: Joon Sung Park
paper_authors._id: 654c58c7471c76068ca4e44b
paper_authors.name: Agrim Gupta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:17:33.515Z
paper_authors._id: 63defb9015266dd945f9f772
paper_authors.avatarUrl: /avatars/b304ac6c3b45553358f8b6cd5c3df293.svg
paper_authors.fullname: Agrim
paper_authors.type: user
paper_authors.user: agrimgupta
paper_authors._id: 654c58c7471c76068ca4e44c
paper_authors.name: Yunzhi Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:17:45.822Z
paper_authors._id: 63793aee2cb50dda39d6e73b
paper_authors.avatarUrl: /avatars/1d7eb1a19ce2e18802e9481200eb03f7.svg
paper_authors.fullname: Yunzhi Zhang
paper_authors.type: user
paper_authors.user: Yunzhi
paper_authors._id: 654c58c7471c76068ca4e44d
paper_authors.name: Deepak Narayanan
paper_authors._id: 654c58c7471c76068ca4e44e
paper_authors.name: Hannah Benita Teufel
paper_authors._id: 654c58c7471c76068ca4e44f
paper_authors.name: Marco Bellagente
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:18:44.191Z
paper_authors._id: 63519eb7a8e595171ab179cb
paper_authors.avatarUrl: /avatars/792789acb90450962c9cdbec5b24e963.svg
paper_authors.fullname: Marco Bellagente
paper_authors.type: user
paper_authors.user: marcob
paper_authors._id: 654c58c7471c76068ca4e450
paper_authors.name: Minguk Kang
paper_authors._id: 654c58c7471c76068ca4e451
paper_authors.name: Taesung Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:19:18.987Z
paper_authors._id: 637563c59ef4e3b87a60d503
paper_authors.avatarUrl: /avatars/8264de115e6305e4720c78fc7dce7f8d.svg
paper_authors.fullname: Taesung Park
paper_authors.type: user
paper_authors.user: taesungp
paper_authors._id: 654c58c7471c76068ca4e452
paper_authors.name: Jure Leskovec
paper_authors._id: 654c58c7471c76068ca4e453
paper_authors.name: Jun-Yan Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:19:31.308Z
paper_authors._id: 63a0acc32fabbbb899952a2b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671474335794-noauth.jpeg
paper_authors.fullname: Jun-Yan Zhu
paper_authors.type: user
paper_authors.user: junyanz
paper_authors._id: 654c58c7471c76068ca4e454
paper_authors.name: Li Fei-Fei
paper_authors._id: 654c58c7471c76068ca4e455
paper_authors.name: Jiajun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:19:56.314Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_authors._id: 654c58c7471c76068ca4e456
paper_authors.name: Stefano Ermon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:20:04.971Z
paper_authors._id: 62f6e244329d4d014d1f4ac5
paper_authors.avatarUrl: /avatars/5a8b2bb063c2ebc340504b22530f6811.svg
paper_authors.fullname: Stefano Ermon
paper_authors.type: user
paper_authors.user: ermonste
paper_authors._id: 654c58c7471c76068ca4e457
paper_authors.name: Percy Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:20:13.067Z
paper_authors._id: 6409651b9e9f790c905b2335
paper_authors.avatarUrl: /avatars/1fb8c80b60f21f65a0a027319101f236.svg
paper_authors.fullname: Percy Liang
paper_authors.type: user
paper_authors.user: percyliang
paper_title: Holistic Evaluation of Text-To-Image Models
paper_summary: The stunning qualitative improvement of recent text-to-image models has led
  to their widespread attention and adoption. However, we lack a comprehensive
  quantitative understanding of their capabilities and risks. To fill this gap,
  we introduce a new benchmark, Holistic Evaluation of Text-to-Image Models
  (HEIM). Whereas previous evaluations focus mostly on text-image alignment and
  image quality, we identify 12 aspects, including text-image alignment, image
  quality, aesthetics, originality, reasoning, knowledge, bias, toxicity,
  fairness, robustness, multilinguality, and efficiency. We curate 62 scenarios
  encompassing these aspects and evaluate 26 state-of-the-art text-to-image
  models on this benchmark. Our results reveal that no single model excels in all
  aspects, with different models demonstrating different strengths. We release
  the generated images and human evaluation results for full transparency at
  https://crfm.stanford.edu/heim/v1.1.0 and the code at
  https://github.com/stanford-crfm/helm, which is integrated with the HELM
  codebase.
paper_ai_summary: A new benchmark evaluates text-to-image models across 12 aspects, revealing no single model excels comprehensively in all areas.
paper_ai_keywords: text-to-image models Holistic Evaluation of Text-to-Image Models HEIM text-image alignment image quality aesthetics originality reasoning knowledge bias toxicity fairness robustness multilinguality efficiency