title: Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic
    Image Design and Generation
summary: We introduce ``Idea to Image,'' a system that enables multimodal iterative
  self-refinement with GPT-4V(ision) for automatic image design and generation.
  Humans can quickly identify the characteristics of different text-to-image
  (T2I) models via iterative explorations. This enables them to efficiently
  convert their high-level generation ideas into effective T2I prompts that can
  produce good images. We investigate if systems based on large multimodal models
  (LMMs) can develop analogous multimodal self-refinement abilities that enable
  exploring unknown models or environments via self-refining tries. Idea2Img
  cyclically generates revised T2I prompts to synthesize draft images, and
  provides directional feedback for prompt revision, both conditioned on its
  memory of the probed T2I model's characteristics. The iterative self-refinement
  brings Idea2Img various advantages over vanilla T2I models. Notably, Idea2Img
  can process input ideas with interleaved image-text sequences, follow ideas
  with design instructions, and generate images of better semantic and visual
  qualities. The user preference study validates the efficacy of multimodal
  iterative self-refinement on automatic image design and generation.
paper_authors._id: 65294587019a20ed658ecc38
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T16:03:49.250Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 65294587019a20ed658ecc39
paper_authors.name: Jianfeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T16:04:13.692Z
paper_authors._id: 641404e74e5305c14f269b3c
paper_authors.avatarUrl: /avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
paper_authors.fullname: Jianfeng Wang
paper_authors.type: user
paper_authors.user: amsword8
paper_authors._id: 65294587019a20ed658ecc3a
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T16:04:28.570Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 65294587019a20ed658ecc3b
paper_authors.name: Kevin Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-16T07:24:49.165Z
paper_authors._id: 6298fd95b58e71e2ac9f3ad8
paper_authors.avatarUrl: /avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
paper_authors.fullname: Kevin Lin
paper_authors.type: user
paper_authors.user: kevinlin311tw
paper_authors._id: 65294587019a20ed658ecc3c
paper_authors.name: Chung-Ching Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:33:21.391Z
paper_authors._id: 6669f406d21cf7d39b1d98ba
paper_authors.avatarUrl: /avatars/5ec270cb8bcb68786379d8a8bc411aaa.svg
paper_authors.fullname: CC Lin
paper_authors.type: user
paper_authors.user: cclin10
paper_authors._id: 65294587019a20ed658ecc3d
paper_authors.name: Zicheng Liu
paper_authors._id: 65294587019a20ed658ecc3e
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T16:05:31.275Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_title: Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic
    Image Design and Generation
paper_summary: We introduce ``Idea to Image,'' a system that enables multimodal iterative
  self-refinement with GPT-4V(ision) for automatic image design and generation.
  Humans can quickly identify the characteristics of different text-to-image
  (T2I) models via iterative explorations. This enables them to efficiently
  convert their high-level generation ideas into effective T2I prompts that can
  produce good images. We investigate if systems based on large multimodal models
  (LMMs) can develop analogous multimodal self-refinement abilities that enable
  exploring unknown models or environments via self-refining tries. Idea2Img
  cyclically generates revised T2I prompts to synthesize draft images, and
  provides directional feedback for prompt revision, both conditioned on its
  memory of the probed T2I model's characteristics. The iterative self-refinement
  brings Idea2Img various advantages over vanilla T2I models. Notably, Idea2Img
  can process input ideas with interleaved image-text sequences, follow ideas
  with design instructions, and generate images of better semantic and visual
  qualities. The user preference study validates the efficacy of multimodal
  iterative self-refinement on automatic image design and generation.
paper_ai_summary: Idea to Image uses iterative multimodal refinement with GPT-4V to enhance text-to-image generation by refining prompts based on model characteristics, leading to higher-quality images.
paper_ai_keywords: GPT-4V multimodal iterative self-refinement text-to-image (T2I) models self-refining trials interleaved image-text sequences design instructions