Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free
  Real Image Editing
Despite recent advances in large-scale text-to-image generative models,
manipulating real images with these models remains a challenging problem. The
main limitations of existing editing methods are that they either fail to
perform with consistent quality on a wide range of image edits or require
time-consuming hyperparameter tuning or fine-tuning of the diffusion model to
preserve the image-specific appearance of the input image. We propose a novel
approach that is built upon a modified diffusion sampling process via the
guidance mechanism. In this work, we explore the self-guidance technique to
preserve the overall structure of the input image and its local regions
appearance that should not be edited. In particular, we explicitly introduce
layout-preserving energy functions that are aimed to save local and global
structures of the source image. Additionally, we propose a noise rescaling
mechanism that allows to preserve noise distribution by balancing the norms of
classifier-free guidance and our proposed guiders during generation. Such a
guiding approach does not require fine-tuning the diffusion model and exact
inversion process. As a result, the proposed method provides a fast and
high-quality editing mechanism. In our experiments, we show through human
evaluation and quantitative analysis that the proposed method allows to produce
desired editing which is more preferable by humans and also achieves a better
trade-off between editing quality and preservation of the original image. Our
code is available at https://github.com/FusionBrainLab/Guide-and-Rescale.
66d8408b6ff4d323370c5cd8
Vadim Titov
claimed_verified
2024-09-06T09:06:06.463Z
66a8085f306528869adf91a9
https://cdn-avatars.huggingface.co/v1/production/uploads/66a8085f306528869adf91a9/UpnEthxKTfWKkSiMTWPpT.jpeg
Vadim Titov
user
macderru
66d8408b6ff4d323370c5cd9
Madina Khalmatova
66d8408b6ff4d323370c5cda
Alexandra Ivanova
admin_assigned
2024-09-06T09:50:34.686Z
6241b14fc44ab21d46c97471
/avatars/b0cdbed2c2ef49ef8f497060b239aa19.svg
Alexandra Ivanova
user
sashapff
66d8408b6ff4d323370c5cdb
Dmitry Vetrov
66d8408b6ff4d323370c5cdc
Aibek Alanov
admin_assigned
2024-09-06T09:50:43.823Z
66680c6451545a8b46c6fd21
/avatars/03707f5ea4e2aa8dc825a9782b00ed85.svg
Aibek Alanov
user
ai-alanov
Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free
  Real Image Editing
Despite recent advances in large-scale text-to-image generative models,
manipulating real images with these models remains a challenging problem. The
main limitations of existing editing methods are that they either fail to
perform with consistent quality on a wide range of image edits or require
time-consuming hyperparameter tuning or fine-tuning of the diffusion model to
preserve the image-specific appearance of the input image. We propose a novel
approach that is built upon a modified diffusion sampling process via the
guidance mechanism. In this work, we explore the self-guidance technique to
preserve the overall structure of the input image and its local regions
appearance that should not be edited. In particular, we explicitly introduce
layout-preserving energy functions that are aimed to save local and global
structures of the source image. Additionally, we propose a noise rescaling
mechanism that allows to preserve noise distribution by balancing the norms of
classifier-free guidance and our proposed guiders during generation. Such a
guiding approach does not require fine-tuning the diffusion model and exact
inversion process. As a result, the proposed method provides a fast and
high-quality editing mechanism. In our experiments, we show through human
evaluation and quantitative analysis that the proposed method allows to produce
desired editing which is more preferable by humans and also achieves a better
trade-off between editing quality and preservation of the original image. Our
code is available at https://github.com/FusionBrainLab/Guide-and-Rescale.
A modified diffusion sampling process using self-guidance and noise rescaling achieves high-quality image editing with preserved structure and appearance without requiring diffusion model fine-tuning.
diffusion sampling process guidance mechanism self-guidance layout-preserving energy functions noise rescaling classifier-free guidance diffusion model fine-tuning human evaluation quantitative analysis