DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image
  Editing
Large-scale Text-to-Image (T2I) diffusion models have revolutionized image
generation over the last few years. Although owning diverse and high-quality
generation capabilities, translating these abilities to fine-grained image
editing remains challenging. In this paper, we propose DiffEditor to rectify
two weaknesses in existing diffusion-based image editing: (1) in complex
scenarios, editing results often lack editing accuracy and exhibit unexpected
artifacts; (2) lack of flexibility to harmonize editing operations, e.g.,
imagine new content. In our solution, we introduce image prompts in
fine-grained image editing, cooperating with the text prompt to better describe
the editing content. To increase the flexibility while maintaining content
consistency, we locally combine stochastic differential equation (SDE) into the
ordinary differential equation (ODE) sampling. In addition, we incorporate
regional score-based gradient guidance and a time travel strategy into the
diffusion sampling, further improving the editing quality. Extensive
experiments demonstrate that our method can efficiently achieve
state-of-the-art performance on various fine-grained image editing tasks,
including editing within a single image (e.g., object moving, resizing, and
content dragging) and across images (e.g., appearance replacing and object
pasting). Our source code is released at
https://github.com/MC-E/DragonDiffusion.
65c1a36355c4f06fa96aa3fb
Chong Mou
65c1a36355c4f06fa96aa3fc
Xintao Wang
65c1a36355c4f06fa96aa3fd
Jiechong Song
65c1a36355c4f06fa96aa3fe
Ying Shan
65c1a36355c4f06fa96aa3ff
Jian Zhang
DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image
  Editing
Large-scale Text-to-Image (T2I) diffusion models have revolutionized image
generation over the last few years. Although owning diverse and high-quality
generation capabilities, translating these abilities to fine-grained image
editing remains challenging. In this paper, we propose DiffEditor to rectify
two weaknesses in existing diffusion-based image editing: (1) in complex
scenarios, editing results often lack editing accuracy and exhibit unexpected
artifacts; (2) lack of flexibility to harmonize editing operations, e.g.,
imagine new content. In our solution, we introduce image prompts in
fine-grained image editing, cooperating with the text prompt to better describe
the editing content. To increase the flexibility while maintaining content
consistency, we locally combine stochastic differential equation (SDE) into the
ordinary differential equation (ODE) sampling. In addition, we incorporate
regional score-based gradient guidance and a time travel strategy into the
diffusion sampling, further improving the editing quality. Extensive
experiments demonstrate that our method can efficiently achieve
state-of-the-art performance on various fine-grained image editing tasks,
including editing within a single image (e.g., object moving, resizing, and
content dragging) and across images (e.g., appearance replacing and object
pasting). Our source code is released at
https://github.com/MC-E/DragonDiffusion.
DiffEditor, a diffusion-based method, improves fine-grained image editing accuracy and flexibility through image prompts, SDE-ODE combination, regional score-based gradient guidance, and time travel strategy.
diffusion models fine-grained image editing image prompts text prompt stochastic differential equation ordinary differential equation regional score-based gradient guidance time travel strategy