SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions
The remarkable capabilities of pretrained image diffusion models have been
utilized not only for generating fixed-size images but also for creating
panoramas. However, naive stitching of multiple images often results in visible
seams. Recent techniques have attempted to address this issue by performing
joint diffusions in multiple windows and averaging latent features in
overlapping regions. However, these approaches, which focus on seamless montage
generation, often yield incoherent outputs by blending different scenes within
a single image. To overcome this limitation, we propose SyncDiffusion, a
plug-and-play module that synchronizes multiple diffusions through gradient
descent from a perceptual similarity loss. Specifically, we compute the
gradient of the perceptual loss using the predicted denoised images at each
denoising step, providing meaningful guidance for achieving coherent montages.
Our experimental results demonstrate that our method produces significantly
more coherent outputs compared to previous methods (66.35% vs. 33.65% in our
user study) while still maintaining fidelity (as assessed by GIQA) and
compatibility with the input prompt (as measured by CLIP score).
6482b06a474d4d463d4c5342
Yuseung Lee
claimed_verified
2023-06-12T09:12:20.366Z
6342796a0875f2c99cfd313b
/avatars/98575092404c4197b20c929a6499a015.svg
Yuseung "Phillip" Lee
user
phillipinseoul
6482b06a474d4d463d4c5343
Kunho Kim
admin_assigned
2023-06-12T13:27:21.011Z
644b5c969279988e0cbecd56
/avatars/be2983528677d88647e031538f3f6c40.svg
Kunho Kim
user
Kunho
6482b06a474d4d463d4c5344
Hyunjin Kim
6482b06a474d4d463d4c5345
Minhyuk Sung
admin_assigned
2023-06-12T13:27:45.261Z
631f432b5ba8c026340a7890
https://cdn-avatars.huggingface.co/v1/production/uploads/631f432b5ba8c026340a7890/9PK7A_TRMpugwYjCsNBf1.jpeg
Minhyuk Sung
user
Minhyuk
SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions
The remarkable capabilities of pretrained image diffusion models have been
utilized not only for generating fixed-size images but also for creating
panoramas. However, naive stitching of multiple images often results in visible
seams. Recent techniques have attempted to address this issue by performing
joint diffusions in multiple windows and averaging latent features in
overlapping regions. However, these approaches, which focus on seamless montage
generation, often yield incoherent outputs by blending different scenes within
a single image. To overcome this limitation, we propose SyncDiffusion, a
plug-and-play module that synchronizes multiple diffusions through gradient
descent from a perceptual similarity loss. Specifically, we compute the
gradient of the perceptual loss using the predicted denoised images at each
denoising step, providing meaningful guidance for achieving coherent montages.
Our experimental results demonstrate that our method produces significantly
more coherent outputs compared to previous methods (66.35% vs. 33.65% in our
user study) while still maintaining fidelity (as assessed by GIQA) and
compatibility with the input prompt (as measured by CLIP score).
SyncDiffusion synchronizes multiple diffusions via perceptual similarity loss to generate coherent panoramic images without visible seams and maintains fidelity and compatibility with input prompts.
pretrained image diffusion models panoramas naive stitching seamless montage generation joint diffusions latent features perceptual similarity loss gradient descent denoised images GIQA CLIP score