title: Controllable Text-to-Image Generation with GPT-4
summary: Current text-to-image generation models often struggle to follow textual
  instructions, especially the ones requiring spatial reasoning. On the other
  hand, Large Language Models (LLMs), such as GPT-4, have shown remarkable
  precision in generating code snippets for sketching out text inputs
  graphically, e.g., via TikZ. In this work, we introduce Control-GPT to guide
  the diffusion-based text-to-image pipelines with programmatic sketches
  generated by GPT-4, enhancing their abilities for instruction following.
  Control-GPT works by querying GPT-4 to write TikZ code, and the generated
  sketches are used as references alongside the text instructions for diffusion
  models (e.g., ControlNet) to generate photo-realistic images. One major
  challenge to training our pipeline is the lack of a dataset containing aligned
  text, images, and sketches. We address the issue by converting instance masks
  in existing datasets into polygons to mimic the sketches used at test time. As
  a result, Control-GPT greatly boosts the controllability of image generation.
  It establishes a new state-of-art on the spatial arrangement and object
  positioning generation and enhances users' control of object positions, sizes,
  etc., nearly doubling the accuracy of prior models. Our work, as a first
  attempt, shows the potential for employing LLMs to enhance the performance in
  computer vision tasks.
paper_authors._id: 64769a94288784ce9abc35b9
paper_authors.name: Tianjun Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-01T08:14:19.658Z
paper_authors._id: 6374cd6b6ea8da14f8fef8dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6374cd6b6ea8da14f8fef8dc/l13bg0tKDjCnUw3I895QZ.png
paper_authors.fullname: Tianjun Zhang
paper_authors.type: user
paper_authors.user: tianjunz
paper_authors._id: 64769a94288784ce9abc35ba
paper_authors.name: Yi Zhang
paper_authors._id: 64769a94288784ce9abc35bb
paper_authors.name: Vibhav Vineet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T12:36:23.726Z
paper_authors._id: 63c8527becdb7c9fdd9cacc6
paper_authors.avatarUrl: /avatars/c8a3f5e1e5159ae5ead41bd9fc2b9b34.svg
paper_authors.fullname: Vibhav Vineet
paper_authors.type: user
paper_authors.user: vibhav-vineet
paper_authors._id: 64769a94288784ce9abc35bc
paper_authors.name: Neel Joshi
paper_authors._id: 64769a94288784ce9abc35bd
paper_authors.name: Xin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-09T18:39:41.291Z
paper_authors._id: 64679a226192d39142245e5e
paper_authors.avatarUrl: /avatars/05abee0b6317f100923936ca2099e9eb.svg
paper_authors.fullname: Xin Eric Wang
paper_authors.type: user
paper_authors.user: xw-eric
paper_title: Controllable Text-to-Image Generation with GPT-4
paper_summary: Current text-to-image generation models often struggle to follow textual
  instructions, especially the ones requiring spatial reasoning. On the other
  hand, Large Language Models (LLMs), such as GPT-4, have shown remarkable
  precision in generating code snippets for sketching out text inputs
  graphically, e.g., via TikZ. In this work, we introduce Control-GPT to guide
  the diffusion-based text-to-image pipelines with programmatic sketches
  generated by GPT-4, enhancing their abilities for instruction following.
  Control-GPT works by querying GPT-4 to write TikZ code, and the generated
  sketches are used as references alongside the text instructions for diffusion
  models (e.g., ControlNet) to generate photo-realistic images. One major
  challenge to training our pipeline is the lack of a dataset containing aligned
  text, images, and sketches. We address the issue by converting instance masks
  in existing datasets into polygons to mimic the sketches used at test time. As
  a result, Control-GPT greatly boosts the controllability of image generation.
  It establishes a new state-of-art on the spatial arrangement and object
  positioning generation and enhances users' control of object positions, sizes,
  etc., nearly doubling the accuracy of prior models. Our work, as a first
  attempt, shows the potential for employing LLMs to enhance the performance in
  computer vision tasks.
paper_ai_summary: Control-GPT enhances text-to-image generation by using GPT-4-generated TikZ sketches as guides, improving spatial arrangement and object positioning accuracy.
paper_ai_keywords: diffusion-based text-to-image ControlNet TikZ instance masks polygons graphical sketches Large Language Models LLMs control of image generation spatial reasoning object positioning text-to-image