title: LIME: Localized Image Editing via Attention Regularization in Diffusion
    Models
summary: Diffusion models (DMs) have gained prominence due to their ability to
  generate high-quality, varied images, with recent advancements in text-to-image
  generation. The research focus is now shifting towards the controllability of
  DMs. A significant challenge within this domain is localized editing, where
  specific areas of an image are modified without affecting the rest of the
  content. This paper introduces LIME for localized image editing in diffusion
  models that do not require user-specified regions of interest (RoI) or
  additional text input. Our method employs features from pre-trained methods and
  a simple clustering technique to obtain precise semantic segmentation maps.
  Then, by leveraging cross-attention maps, it refines these segments for
  localized edits. Finally, we propose a novel cross-attention regularization
  technique that penalizes unrelated cross-attention scores in the RoI during the
  denoising steps, ensuring localized edits. Our approach, without re-training
  and fine-tuning, consistently improves the performance of existing methods in
  various editing benchmarks.
paper_authors._id: 657bd57c12e4dfc7a938fe7e
paper_authors.name: Enis Simsar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-15T08:04:14.250Z
paper_authors._id: 63412f2add8853dc7e306a4f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/nqbteeQhS2dPA5LVy7dTD.png
paper_authors.fullname: Enis Simsar
paper_authors.type: user
paper_authors.user: enisimsar
paper_authors._id: 657bd57c12e4dfc7a938fe7f
paper_authors.name: Alessio Tonioni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:52:03.537Z
paper_authors._id: 63402b30e670ff9cf63d8caa
paper_authors.avatarUrl: /avatars/0aee84d132a78d4ec71663836a57a245.svg
paper_authors.fullname: Alessio Tonioni
paper_authors.type: user
paper_authors.user: Alessiot
paper_authors._id: 657bd57c12e4dfc7a938fe80
paper_authors.name: Yongqin Xian
paper_authors._id: 657bd57c12e4dfc7a938fe81
paper_authors.name: Thomas Hofmann
paper_authors._id: 657bd57c12e4dfc7a938fe82
paper_authors.name: Federico Tombari
paper_title: LIME: Localized Image Editing via Attention Regularization in Diffusion
    Models
paper_summary: Diffusion models (DMs) have gained prominence due to their ability to
  generate high-quality, varied images, with recent advancements in text-to-image
  generation. The research focus is now shifting towards the controllability of
  DMs. A significant challenge within this domain is localized editing, where
  specific areas of an image are modified without affecting the rest of the
  content. This paper introduces LIME for localized image editing in diffusion
  models that do not require user-specified regions of interest (RoI) or
  additional text input. Our method employs features from pre-trained methods and
  a simple clustering technique to obtain precise semantic segmentation maps.
  Then, by leveraging cross-attention maps, it refines these segments for
  localized edits. Finally, we propose a novel cross-attention regularization
  technique that penalizes unrelated cross-attention scores in the RoI during the
  denoising steps, ensuring localized edits. Our approach, without re-training
  and fine-tuning, consistently improves the performance of existing methods in
  various editing benchmarks.
paper_ai_summary: LIME enhances localized image editing in diffusion models using pre-trained features, clustering, refined cross-attention, and regularization without re-training or fine-tuning.
paper_ai_keywords: diffusion models LIME localized editing pre-trained methods clustering cross-attention maps cross-attention regularization denoising steps