title: Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free
    Real Image Editing
summary: Despite recent advances in large-scale text-to-image generative models,
  manipulating real images with these models remains a challenging problem. The
  main limitations of existing editing methods are that they either fail to
  perform with consistent quality on a wide range of image edits or require
  time-consuming hyperparameter tuning or fine-tuning of the diffusion model to
  preserve the image-specific appearance of the input image. We propose a novel
  approach that is built upon a modified diffusion sampling process via the
  guidance mechanism. In this work, we explore the self-guidance technique to
  preserve the overall structure of the input image and its local regions
  appearance that should not be edited. In particular, we explicitly introduce
  layout-preserving energy functions that are aimed to save local and global
  structures of the source image. Additionally, we propose a noise rescaling
  mechanism that allows to preserve noise distribution by balancing the norms of
  classifier-free guidance and our proposed guiders during generation. Such a
  guiding approach does not require fine-tuning the diffusion model and exact
  inversion process. As a result, the proposed method provides a fast and
  high-quality editing mechanism. In our experiments, we show through human
  evaluation and quantitative analysis that the proposed method allows to produce
  desired editing which is more preferable by humans and also achieves a better
  trade-off between editing quality and preservation of the original image. Our
  code is available at https://github.com/FusionBrainLab/Guide-and-Rescale.
paper_authors._id: 66d8408b6ff4d323370c5cd8
paper_authors.name: Vadim Titov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-06T09:06:06.463Z
paper_authors._id: 66a8085f306528869adf91a9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66a8085f306528869adf91a9/UpnEthxKTfWKkSiMTWPpT.jpeg
paper_authors.fullname: Vadim Titov
paper_authors.type: user
paper_authors.user: macderru
paper_authors._id: 66d8408b6ff4d323370c5cd9
paper_authors.name: Madina Khalmatova
paper_authors._id: 66d8408b6ff4d323370c5cda
paper_authors.name: Alexandra Ivanova
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-06T09:50:34.686Z
paper_authors._id: 6241b14fc44ab21d46c97471
paper_authors.avatarUrl: /avatars/b0cdbed2c2ef49ef8f497060b239aa19.svg
paper_authors.fullname: Alexandra Ivanova
paper_authors.type: user
paper_authors.user: sashapff
paper_authors._id: 66d8408b6ff4d323370c5cdb
paper_authors.name: Dmitry Vetrov
paper_authors._id: 66d8408b6ff4d323370c5cdc
paper_authors.name: Aibek Alanov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-06T09:50:43.823Z
paper_authors._id: 66680c6451545a8b46c6fd21
paper_authors.avatarUrl: /avatars/03707f5ea4e2aa8dc825a9782b00ed85.svg
paper_authors.fullname: Aibek Alanov
paper_authors.type: user
paper_authors.user: ai-alanov
paper_title: Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free
    Real Image Editing
paper_summary: Despite recent advances in large-scale text-to-image generative models,
  manipulating real images with these models remains a challenging problem. The
  main limitations of existing editing methods are that they either fail to
  perform with consistent quality on a wide range of image edits or require
  time-consuming hyperparameter tuning or fine-tuning of the diffusion model to
  preserve the image-specific appearance of the input image. We propose a novel
  approach that is built upon a modified diffusion sampling process via the
  guidance mechanism. In this work, we explore the self-guidance technique to
  preserve the overall structure of the input image and its local regions
  appearance that should not be edited. In particular, we explicitly introduce
  layout-preserving energy functions that are aimed to save local and global
  structures of the source image. Additionally, we propose a noise rescaling
  mechanism that allows to preserve noise distribution by balancing the norms of
  classifier-free guidance and our proposed guiders during generation. Such a
  guiding approach does not require fine-tuning the diffusion model and exact
  inversion process. As a result, the proposed method provides a fast and
  high-quality editing mechanism. In our experiments, we show through human
  evaluation and quantitative analysis that the proposed method allows to produce
  desired editing which is more preferable by humans and also achieves a better
  trade-off between editing quality and preservation of the original image. Our
  code is available at https://github.com/FusionBrainLab/Guide-and-Rescale.
paper_ai_summary: A modified diffusion sampling process using self-guidance and noise rescaling achieves high-quality image editing with preserved structure and appearance without requiring diffusion model fine-tuning.
paper_ai_keywords: diffusion sampling process guidance mechanism self-guidance layout-preserving energy functions noise rescaling classifier-free guidance diffusion model fine-tuning human evaluation quantitative analysis