title: Diffusion Self-Guidance for Controllable Image Generation
summary: Large-scale generative models are capable of producing high-quality images
  from detailed text descriptions. However, many aspects of an image are
  difficult or impossible to convey through text. We introduce self-guidance, a
  method that provides greater control over generated images by guiding the
  internal representations of diffusion models. We demonstrate that properties
  such as the shape, location, and appearance of objects can be extracted from
  these representations and used to steer sampling. Self-guidance works similarly
  to classifier guidance, but uses signals present in the pretrained model
  itself, requiring no additional models or training. We show how a simple set of
  properties can be composed to perform challenging image manipulations, such as
  modifying the position or size of objects, merging the appearance of objects in
  one image with the layout of another, composing objects from many images into
  one, and more. We also show that self-guidance can be used to edit real images.
  For results and an interactive demo, see our project page at
  https://dave.ml/selfguidance/
paper_authors._id: 64794aaca68454566355bbc8
paper_authors.name: Dave Epstein
paper_authors._id: 64794aaca68454566355bbc9
paper_authors.name: Allan Jabri
paper_authors._id: 64794aaca68454566355bbca
paper_authors.name: Ben Poole
paper_authors._id: 64794aaca68454566355bbcb
paper_authors.name: Alexei A. Efros
paper_authors._id: 64794aaca68454566355bbcc
paper_authors.name: Aleksander Holynski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T12:56:22.480Z
paper_authors._id: 62f6e2792e53c2efd33faa92
paper_authors.avatarUrl: /avatars/85d94b4022577747b8d2d10a82c2f3c7.svg
paper_authors.fullname: Aleksander Holynski
paper_authors.type: user
paper_authors.user: holynski
paper_title: Diffusion Self-Guidance for Controllable Image Generation
paper_summary: Large-scale generative models are capable of producing high-quality images
  from detailed text descriptions. However, many aspects of an image are
  difficult or impossible to convey through text. We introduce self-guidance, a
  method that provides greater control over generated images by guiding the
  internal representations of diffusion models. We demonstrate that properties
  such as the shape, location, and appearance of objects can be extracted from
  these representations and used to steer sampling. Self-guidance works similarly
  to classifier guidance, but uses signals present in the pretrained model
  itself, requiring no additional models or training. We show how a simple set of
  properties can be composed to perform challenging image manipulations, such as
  modifying the position or size of objects, merging the appearance of objects in
  one image with the layout of another, composing objects from many images into
  one, and more. We also show that self-guidance can be used to edit real images.
  For results and an interactive demo, see our project page at
  https://dave.ml/selfguidance/
paper_ai_keywords: generative models diffusion models self-guidance internal representations shape location appearance classifier guidance large-scale visualizations samples manipulations composition real images