title: LooseControl: Lifting ControlNet for Generalized Depth Conditioning
summary: We present LooseControl to allow generalized depth conditioning for
  diffusion-based image generation. ControlNet, the SOTA for depth-conditioned
  image generation, produces remarkable results but relies on having access to
  detailed depth maps for guidance. Creating such exact depth maps, in many
  scenarios, is challenging. This paper introduces a generalized version of depth
  conditioning that enables many new content-creation workflows. Specifically, we
  allow (C1) scene boundary control for loosely specifying scenes with only
  boundary conditions, and (C2) 3D box control for specifying layout locations of
  the target objects rather than the exact shape and appearance of the objects.
  Using LooseControl, along with text guidance, users can create complex
  environments (e.g., rooms, street views, etc.) by specifying only scene
  boundaries and locations of primary objects. Further, we provide two editing
  mechanisms to refine the results: (E1) 3D box editing enables the user to
  refine images by changing, adding, or removing boxes while freezing the style
  of the image. This yields minimal changes apart from changes induced by the
  edited boxes. (E2) Attribute editing proposes possible editing directions to
  change one particular aspect of the scene, such as the overall object density
  or a particular object. Extensive tests and comparisons with baselines
  demonstrate the generality of our method. We believe that LooseControl can
  become an important design tool for easily creating complex environments and be
  extended to other forms of guidance channels. Code and more information are
  available at https://shariqfarooq123.github.io/loose-control/ .
paper_authors._id: 65712af9f3853f99bc43355d
paper_authors.name: Shariq Farooq Bhat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:16:28.744Z
paper_authors._id: 63fcae4f987f631186e50fba
paper_authors.avatarUrl: /avatars/8ac9dff40d1b483b4f2f4e4cbd3f9088.svg
paper_authors.fullname: Shariq Farooq Bhat
paper_authors.type: user
paper_authors.user: shariqfarooq
paper_authors._id: 65712af9f3853f99bc43355e
paper_authors.name: Niloy J. Mitra
paper_authors._id: 65712af9f3853f99bc43355f
paper_authors.name: Peter Wonka
paper_title: LooseControl: Lifting ControlNet for Generalized Depth Conditioning
paper_summary: We present LooseControl to allow generalized depth conditioning for
  diffusion-based image generation. ControlNet, the SOTA for depth-conditioned
  image generation, produces remarkable results but relies on having access to
  detailed depth maps for guidance. Creating such exact depth maps, in many
  scenarios, is challenging. This paper introduces a generalized version of depth
  conditioning that enables many new content-creation workflows. Specifically, we
  allow (C1) scene boundary control for loosely specifying scenes with only
  boundary conditions, and (C2) 3D box control for specifying layout locations of
  the target objects rather than the exact shape and appearance of the objects.
  Using LooseControl, along with text guidance, users can create complex
  environments (e.g., rooms, street views, etc.) by specifying only scene
  boundaries and locations of primary objects. Further, we provide two editing
  mechanisms to refine the results: (E1) 3D box editing enables the user to
  refine images by changing, adding, or removing boxes while freezing the style
  of the image. This yields minimal changes apart from changes induced by the
  edited boxes. (E2) Attribute editing proposes possible editing directions to
  change one particular aspect of the scene, such as the overall object density
  or a particular object. Extensive tests and comparisons with baselines
  demonstrate the generality of our method. We believe that LooseControl can
  become an important design tool for easily creating complex environments and be
  extended to other forms of guidance channels. Code and more information are
  available at https://shariqfarooq123.github.io/loose-control/ .
paper_ai_keywords: diffusion-based image generation ControlNet depth-conditioned image generation depth maps generalized depth conditioning scene boundary control 3D box control text guidance 3D box editing attribute editing object density