LIME: Localized Image Editing via Attention Regularization in Diffusion
  Models
Diffusion models (DMs) have gained prominence due to their ability to
generate high-quality, varied images, with recent advancements in text-to-image
generation. The research focus is now shifting towards the controllability of
DMs. A significant challenge within this domain is localized editing, where
specific areas of an image are modified without affecting the rest of the
content. This paper introduces LIME for localized image editing in diffusion
models that do not require user-specified regions of interest (RoI) or
additional text input. Our method employs features from pre-trained methods and
a simple clustering technique to obtain precise semantic segmentation maps.
Then, by leveraging cross-attention maps, it refines these segments for
localized edits. Finally, we propose a novel cross-attention regularization
technique that penalizes unrelated cross-attention scores in the RoI during the
denoising steps, ensuring localized edits. Our approach, without re-training
and fine-tuning, consistently improves the performance of existing methods in
various editing benchmarks.
657bd57c12e4dfc7a938fe7e
Enis Simsar
claimed_verified
2023-12-15T08:04:14.250Z
63412f2add8853dc7e306a4f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/nqbteeQhS2dPA5LVy7dTD.png
Enis Simsar
user
enisimsar
657bd57c12e4dfc7a938fe7f
Alessio Tonioni
admin_assigned
2023-12-15T10:52:03.537Z
63402b30e670ff9cf63d8caa
/avatars/0aee84d132a78d4ec71663836a57a245.svg
Alessio Tonioni
user
Alessiot
657bd57c12e4dfc7a938fe80
Yongqin Xian
657bd57c12e4dfc7a938fe81
Thomas Hofmann
657bd57c12e4dfc7a938fe82
Federico Tombari
LIME: Localized Image Editing via Attention Regularization in Diffusion
  Models
Diffusion models (DMs) have gained prominence due to their ability to
generate high-quality, varied images, with recent advancements in text-to-image
generation. The research focus is now shifting towards the controllability of
DMs. A significant challenge within this domain is localized editing, where
specific areas of an image are modified without affecting the rest of the
content. This paper introduces LIME for localized image editing in diffusion
models that do not require user-specified regions of interest (RoI) or
additional text input. Our method employs features from pre-trained methods and
a simple clustering technique to obtain precise semantic segmentation maps.
Then, by leveraging cross-attention maps, it refines these segments for
localized edits. Finally, we propose a novel cross-attention regularization
technique that penalizes unrelated cross-attention scores in the RoI during the
denoising steps, ensuring localized edits. Our approach, without re-training
and fine-tuning, consistently improves the performance of existing methods in
various editing benchmarks.
LIME enhances localized image editing in diffusion models using pre-trained features, clustering, refined cross-attention, and regularization without re-training or fine-tuning.
diffusion models LIME localized editing pre-trained methods clustering cross-attention maps cross-attention regularization denoising steps