title: DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual
    Design
summary: We introduce DEsignBench, a text-to-image (T2I) generation benchmark tailored
  for visual design scenarios. Recent T2I models like DALL-E 3 and others, have
  demonstrated remarkable capabilities in generating photorealistic images that
  align closely with textual inputs. While the allure of creating visually
  captivating images is undeniable, our emphasis extends beyond mere aesthetic
  pleasure. We aim to investigate the potential of using these powerful models in
  authentic design contexts. In pursuit of this goal, we develop DEsignBench,
  which incorporates test samples designed to assess T2I models on both "design
  technical capability" and "design application scenario." Each of these two
  dimensions is supported by a diverse set of specific design categories. We
  explore DALL-E 3 together with other leading T2I models on DEsignBench,
  resulting in a comprehensive visual gallery for side-by-side comparisons. For
  DEsignBench benchmarking, we perform human evaluations on generated images in
  DEsignBench gallery, against the criteria of image-text alignment, visual
  aesthetic, and design creativity. Our evaluation also considers other
  specialized design capabilities, including text rendering, layout composition,
  color harmony, 3D design, and medium style. In addition to human evaluations,
  we introduce the first automatic image generation evaluator powered by GPT-4V.
  This evaluator provides ratings that align well with human judgments, while
  being easily replicable and cost-efficient. A high-resolution version is
  available at
  https://github.com/design-bench/design-bench.github.io/raw/main/designbench.pdf?download=
paper_authors._id: 65373e288cc9d7c02405850c
paper_authors.name: Kevin Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-24T07:29:15.288Z
paper_authors._id: 6298fd95b58e71e2ac9f3ad8
paper_authors.avatarUrl: /avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
paper_authors.fullname: Kevin Lin
paper_authors.type: user
paper_authors.user: kevinlin311tw
paper_authors._id: 65373e288cc9d7c02405850d
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:24:55.698Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 65373e288cc9d7c02405850e
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:37:42.242Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 65373e288cc9d7c02405850f
paper_authors.name: Jianfeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:38:03.293Z
paper_authors._id: 641404e74e5305c14f269b3c
paper_authors.avatarUrl: /avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
paper_authors.fullname: Jianfeng Wang
paper_authors.type: user
paper_authors.user: amsword8
paper_authors._id: 65373e288cc9d7c024058510
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:38:16.143Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_title: DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual
    Design
paper_summary: We introduce DEsignBench, a text-to-image (T2I) generation benchmark tailored
  for visual design scenarios. Recent T2I models like DALL-E 3 and others, have
  demonstrated remarkable capabilities in generating photorealistic images that
  align closely with textual inputs. While the allure of creating visually
  captivating images is undeniable, our emphasis extends beyond mere aesthetic
  pleasure. We aim to investigate the potential of using these powerful models in
  authentic design contexts. In pursuit of this goal, we develop DEsignBench,
  which incorporates test samples designed to assess T2I models on both "design
  technical capability" and "design application scenario." Each of these two
  dimensions is supported by a diverse set of specific design categories. We
  explore DALL-E 3 together with other leading T2I models on DEsignBench,
  resulting in a comprehensive visual gallery for side-by-side comparisons. For
  DEsignBench benchmarking, we perform human evaluations on generated images in
  DEsignBench gallery, against the criteria of image-text alignment, visual
  aesthetic, and design creativity. Our evaluation also considers other
  specialized design capabilities, including text rendering, layout composition,
  color harmony, 3D design, and medium style. In addition to human evaluations,
  we introduce the first automatic image generation evaluator powered by GPT-4V.
  This evaluator provides ratings that align well with human judgments, while
  being easily replicable and cost-efficient. A high-resolution version is
  available at
  https://github.com/design-bench/design-bench.github.io/raw/main/designbench.pdf?download=
paper_ai_summary: DEsignBench evaluates text-to-image models in visual design contexts with human and automatic assessments across various design criteria.
paper_ai_keywords: text-to-image T2I DEsignBench DALL-E 3 image-text alignment visual aesthetic design creativity text rendering layout composition color harmony 3D design medium style GPT-4V