MAG-Edit: Localized Image Editing in Complex Scenarios via
  Mask-Based Attention-Adjusted
  Guidance
Recent diffusion-based image editing approaches have exhibited impressive
editing capabilities in images with simple compositions. However, localized
editing in complex scenarios has not been well-studied in the literature,
despite its growing real-world demands. Existing mask-based inpainting methods
fall short of retaining the underlying structure within the edit region.
Meanwhile, mask-free attention-based methods often exhibit editing leakage and
misalignment in more complex compositions. In this work, we develop
MAG-Edit, a training-free, inference-stage optimization method,
which enables localized image editing in complex scenarios. In particular,
MAG-Edit optimizes the noise latent feature in diffusion models by maximizing
two mask-based cross-attention constraints of the edit token, which in turn
gradually enhances the local alignment with the desired prompt. Extensive
quantitative and qualitative experiments demonstrate the effectiveness of our
method in achieving both text alignment and structure preservation for
localized editing within complex scenarios.
65813b40560c1557cfdc9690
Qi Mao
claimed_verified
2023-12-19T10:16:53.496Z
6388a7e98a5dbe2f3dc61faa
https://cdn-avatars.huggingface.co/v1/production/uploads/6388a7e98a5dbe2f3dc61faa/Zj8pR7BG_wOQjsYzosb5p.jpeg
Qi Mao
user
HelenMao
65813b40560c1557cfdc9691
Lan Chen
claimed_verified
2023-12-19T10:53:47.421Z
640d704c8036cc2142299c19
https://cdn-avatars.huggingface.co/v1/production/uploads/640d704c8036cc2142299c19/Wt9AslcVxWOSSc11epk8l.jpeg
Lan Chen
user
Orannue
65813b40560c1557cfdc9692
Yuchao Gu
admin_assigned
2023-12-19T09:51:21.943Z
63021630a35b21bd8a53305a
/avatars/7a7e8b39749eda61e57d8a1908726558.svg
Gu Yuchao
user
guyuchao
65813b40560c1557cfdc9693
Zhen Fang
claimed_verified
2023-12-19T10:53:55.026Z
64b0a5037a475fba70a7260d
https://cdn-avatars.huggingface.co/v1/production/uploads/64b0a5037a475fba70a7260d/MauBbb6raMA23yrR1Zq21.jpeg
Zhen Fang
user
CostaliyA
65813b40560c1557cfdc9694
Mike Zheng Shou
admin_assigned
2023-12-19T09:50:47.842Z
63a55320ce5763e06f78519c
https://cdn-avatars.huggingface.co/v1/production/uploads/1671779060549-noauth.jpeg
Mike Shou
user
mikeshou
MAG-Edit: Localized Image Editing in Complex Scenarios via
  Mask-Based Attention-Adjusted
  Guidance
Recent diffusion-based image editing approaches have exhibited impressive
editing capabilities in images with simple compositions. However, localized
editing in complex scenarios has not been well-studied in the literature,
despite its growing real-world demands. Existing mask-based inpainting methods
fall short of retaining the underlying structure within the edit region.
Meanwhile, mask-free attention-based methods often exhibit editing leakage and
misalignment in more complex compositions. In this work, we develop
MAG-Edit, a training-free, inference-stage optimization method,
which enables localized image editing in complex scenarios. In particular,
MAG-Edit optimizes the noise latent feature in diffusion models by maximizing
two mask-based cross-attention constraints of the edit token, which in turn
gradually enhances the local alignment with the desired prompt. Extensive
quantitative and qualitative experiments demonstrate the effectiveness of our
method in achieving both text alignment and structure preservation for
localized editing within complex scenarios.
MAG-Edit enhances localized image editing in complex scenarios by optimizing noise latent features in diffusion models through mask-based cross-attention constraints.
diffusion models noise latent feature cross-attention edit token text alignment structure preservation