Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic
  Image Design and Generation
We introduce ``Idea to Image,'' a system that enables multimodal iterative
self-refinement with GPT-4V(ision) for automatic image design and generation.
Humans can quickly identify the characteristics of different text-to-image
(T2I) models via iterative explorations. This enables them to efficiently
convert their high-level generation ideas into effective T2I prompts that can
produce good images. We investigate if systems based on large multimodal models
(LMMs) can develop analogous multimodal self-refinement abilities that enable
exploring unknown models or environments via self-refining tries. Idea2Img
cyclically generates revised T2I prompts to synthesize draft images, and
provides directional feedback for prompt revision, both conditioned on its
memory of the probed T2I model's characteristics. The iterative self-refinement
brings Idea2Img various advantages over vanilla T2I models. Notably, Idea2Img
can process input ideas with interleaved image-text sequences, follow ideas
with design instructions, and generate images of better semantic and visual
qualities. The user preference study validates the efficacy of multimodal
iterative self-refinement on automatic image design and generation.
65294587019a20ed658ecc38
Zhengyuan Yang
admin_assigned
2023-10-13T16:03:49.250Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
65294587019a20ed658ecc39
Jianfeng Wang
admin_assigned
2023-10-13T16:04:13.692Z
641404e74e5305c14f269b3c
/avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
Jianfeng Wang
user
amsword8
65294587019a20ed658ecc3a
Linjie Li
admin_assigned
2023-10-13T16:04:28.570Z
63db16fff03c3d71ef397206
/avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
Linjie Li
user
linjieli222
65294587019a20ed658ecc3b
Kevin Lin
claimed_verified
2023-10-16T07:24:49.165Z
6298fd95b58e71e2ac9f3ad8
/avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
Kevin Lin
user
kevinlin311tw
65294587019a20ed658ecc3c
Chung-Ching Lin
claimed_verified
2024-06-13T09:33:21.391Z
6669f406d21cf7d39b1d98ba
/avatars/5ec270cb8bcb68786379d8a8bc411aaa.svg
CC Lin
user
cclin10
65294587019a20ed658ecc3d
Zicheng Liu
65294587019a20ed658ecc3e
Lijuan Wang
admin_assigned
2023-10-13T16:05:31.275Z
6413521d4e5305c14f22e110
/avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
Lijuan Wang
user
Lijuan
Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic
  Image Design and Generation
We introduce ``Idea to Image,'' a system that enables multimodal iterative
self-refinement with GPT-4V(ision) for automatic image design and generation.
Humans can quickly identify the characteristics of different text-to-image
(T2I) models via iterative explorations. This enables them to efficiently
convert their high-level generation ideas into effective T2I prompts that can
produce good images. We investigate if systems based on large multimodal models
(LMMs) can develop analogous multimodal self-refinement abilities that enable
exploring unknown models or environments via self-refining tries. Idea2Img
cyclically generates revised T2I prompts to synthesize draft images, and
provides directional feedback for prompt revision, both conditioned on its
memory of the probed T2I model's characteristics. The iterative self-refinement
brings Idea2Img various advantages over vanilla T2I models. Notably, Idea2Img
can process input ideas with interleaved image-text sequences, follow ideas
with design instructions, and generate images of better semantic and visual
qualities. The user preference study validates the efficacy of multimodal
iterative self-refinement on automatic image design and generation.
Idea to Image uses iterative multimodal refinement with GPT-4V to enhance text-to-image generation by refining prompts based on model characteristics, leading to higher-quality images.
GPT-4V multimodal iterative self-refinement text-to-image (T2I) models self-refining trials interleaved image-text sequences design instructions