FAM Diffusion: Frequency and Attention Modulation for High-Resolution
  Image Generation with Stable Diffusion
Diffusion models are proficient at generating high-quality images. They are
however effective only when operating at the resolution used during training.
Inference at a scaled resolution leads to repetitive patterns and structural
distortions. Retraining at higher resolutions quickly becomes prohibitive.
Thus, methods enabling pre-existing diffusion models to operate at flexible
test-time resolutions are highly desirable. Previous works suffer from frequent
artifacts and often introduce large latency overheads. We propose two simple
modules that combine to solve these issues. We introduce a Frequency Modulation
(FM) module that leverages the Fourier domain to improve the global structure
consistency, and an Attention Modulation (AM) module which improves the
consistency of local texture patterns, a problem largely ignored in prior
works. Our method, coined Fam diffusion, can seamlessly integrate into any
latent diffusion model and requires no additional training. Extensive
qualitative results highlight the effectiveness of our method in addressing
structural and local artifacts, while quantitative results show
state-of-the-art performance. Also, our method avoids redundant inference
tricks for improved consistency such as patch-based or progressive generation,
leading to negligible latency overheads.
674d8ddeda397a3869dae2ba
Haosen Yang
674d8ddeda397a3869dae2bb
Adrian Bulat
admin_assigned
2024-12-02T22:08:37.877Z
64a6e0923987f4dd3c37087d
/avatars/88bc86ddce1b38a012b07c81f5c61183.svg
Adrian Bulat
user
adrianb1
674d8ddeda397a3869dae2bc
Isma Hadji
admin_assigned
2024-12-02T22:08:32.099Z
638f2f9bc67af472d318a6f0
/avatars/f644218215d4a58c4de43486b96d94de.svg
Isma Hadji
user
hadjisma
674d8ddeda397a3869dae2bd
Hai X. Pham
674d8ddeda397a3869dae2be
Xiatian Zhu
admin_assigned
2024-12-02T22:08:02.924Z
647ef9e7aa8c04bbf9362a8a
/avatars/dc23e15acf7ee7f3b9f1a68c2716a66d.svg
Xiatian Zhu
user
Xiatian-Zhu
674d8ddeda397a3869dae2bf
Georgios Tzimiropoulos
admin_assigned
2024-12-02T22:07:56.398Z
649db0dcd8f9b2e2cb47e4d7
/avatars/e1fb64d5302271a3557ba306a85f2363.svg
Georgios Tzimiropoulos
user
tzimiro
674d8ddeda397a3869dae2c0
Brais Martinez
FAM Diffusion: Frequency and Attention Modulation for High-Resolution
  Image Generation with Stable Diffusion
Diffusion models are proficient at generating high-quality images. They are
however effective only when operating at the resolution used during training.
Inference at a scaled resolution leads to repetitive patterns and structural
distortions. Retraining at higher resolutions quickly becomes prohibitive.
Thus, methods enabling pre-existing diffusion models to operate at flexible
test-time resolutions are highly desirable. Previous works suffer from frequent
artifacts and often introduce large latency overheads. We propose two simple
modules that combine to solve these issues. We introduce a Frequency Modulation
(FM) module that leverages the Fourier domain to improve the global structure
consistency, and an Attention Modulation (AM) module which improves the
consistency of local texture patterns, a problem largely ignored in prior
works. Our method, coined Fam diffusion, can seamlessly integrate into any
latent diffusion model and requires no additional training. Extensive
qualitative results highlight the effectiveness of our method in addressing
structural and local artifacts, while quantitative results show
state-of-the-art performance. Also, our method avoids redundant inference
tricks for improved consistency such as patch-based or progressive generation,
leading to negligible latency overheads.
Fam diffusion integrates Frequency Modulation and Attention Modulation modules to enable high-quality image generation at flexible resolutions without additional training or significant latency.
Frequency Modulation Attention Modulation latent diffusion model