title: GrounDiT: Grounding Diffusion Transformers via Noisy Patch
    Transplantation
summary: We introduce a novel training-free spatial grounding technique for
  text-to-image generation using Diffusion Transformers (DiT). Spatial grounding
  with bounding boxes has gained attention for its simplicity and versatility,
  allowing for enhanced user control in image generation. However, prior
  training-free approaches often rely on updating the noisy image during the
  reverse diffusion process via backpropagation from custom loss functions, which
  frequently struggle to provide precise control over individual bounding boxes.
  In this work, we leverage the flexibility of the Transformer architecture,
  demonstrating that DiT can generate noisy patches corresponding to each
  bounding box, fully encoding the target object and allowing for fine-grained
  control over each region. Our approach builds on an intriguing property of DiT,
  which we refer to as semantic sharing. Due to semantic sharing, when a smaller
  patch is jointly denoised alongside a generatable-size image, the two become
  "semantic clones". Each patch is denoised in its own branch of the generation
  process and then transplanted into the corresponding region of the original
  noisy image at each timestep, resulting in robust spatial grounding for each
  bounding box. In our experiments on the HRS and DrawBench benchmarks, we
  achieve state-of-the-art performance compared to previous training-free spatial
  grounding approaches.
paper_authors._id: 672074e1ddbde77be9cfc529
paper_authors.name: Phillip Y. Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:53:19.885Z
paper_authors._id: 6342796a0875f2c99cfd313b
paper_authors.avatarUrl: /avatars/98575092404c4197b20c929a6499a015.svg
paper_authors.fullname: Yuseung "Phillip" Lee
paper_authors.type: user
paper_authors.user: phillipinseoul
paper_authors._id: 672074e1ddbde77be9cfc52a
paper_authors.name: Taehoon Yoon
paper_authors._id: 672074e1ddbde77be9cfc52b
paper_authors.name: Minhyuk Sung
paper_title: GrounDiT: Grounding Diffusion Transformers via Noisy Patch
    Transplantation
paper_summary: We introduce a novel training-free spatial grounding technique for
  text-to-image generation using Diffusion Transformers (DiT). Spatial grounding
  with bounding boxes has gained attention for its simplicity and versatility,
  allowing for enhanced user control in image generation. However, prior
  training-free approaches often rely on updating the noisy image during the
  reverse diffusion process via backpropagation from custom loss functions, which
  frequently struggle to provide precise control over individual bounding boxes.
  In this work, we leverage the flexibility of the Transformer architecture,
  demonstrating that DiT can generate noisy patches corresponding to each
  bounding box, fully encoding the target object and allowing for fine-grained
  control over each region. Our approach builds on an intriguing property of DiT,
  which we refer to as semantic sharing. Due to semantic sharing, when a smaller
  patch is jointly denoised alongside a generatable-size image, the two become
  "semantic clones". Each patch is denoised in its own branch of the generation
  process and then transplanted into the corresponding region of the original
  noisy image at each timestep, resulting in robust spatial grounding for each
  bounding box. In our experiments on the HRS and DrawBench benchmarks, we
  achieve state-of-the-art performance compared to previous training-free spatial
  grounding approaches.
paper_ai_keywords: Diffusion Transformers (DiT) spatial grounding bounding boxes reverse diffusion process backpropagation custom loss functions Transformer architecture noisy patches target object fine-grained control semantic sharing semantic clones generation process HRS benchmark DrawBench benchmark state-of-the-art performance