title: FAM Diffusion: Frequency and Attention Modulation for High-Resolution
    Image Generation with Stable Diffusion
summary: Diffusion models are proficient at generating high-quality images. They are
  however effective only when operating at the resolution used during training.
  Inference at a scaled resolution leads to repetitive patterns and structural
  distortions. Retraining at higher resolutions quickly becomes prohibitive.
  Thus, methods enabling pre-existing diffusion models to operate at flexible
  test-time resolutions are highly desirable. Previous works suffer from frequent
  artifacts and often introduce large latency overheads. We propose two simple
  modules that combine to solve these issues. We introduce a Frequency Modulation
  (FM) module that leverages the Fourier domain to improve the global structure
  consistency, and an Attention Modulation (AM) module which improves the
  consistency of local texture patterns, a problem largely ignored in prior
  works. Our method, coined Fam diffusion, can seamlessly integrate into any
  latent diffusion model and requires no additional training. Extensive
  qualitative results highlight the effectiveness of our method in addressing
  structural and local artifacts, while quantitative results show
  state-of-the-art performance. Also, our method avoids redundant inference
  tricks for improved consistency such as patch-based or progressive generation,
  leading to negligible latency overheads.
paper_authors._id: 674d8ddeda397a3869dae2ba
paper_authors.name: Haosen Yang
paper_authors._id: 674d8ddeda397a3869dae2bb
paper_authors.name: Adrian Bulat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T22:08:37.877Z
paper_authors._id: 64a6e0923987f4dd3c37087d
paper_authors.avatarUrl: /avatars/88bc86ddce1b38a012b07c81f5c61183.svg
paper_authors.fullname: Adrian Bulat
paper_authors.type: user
paper_authors.user: adrianb1
paper_authors._id: 674d8ddeda397a3869dae2bc
paper_authors.name: Isma Hadji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T22:08:32.099Z
paper_authors._id: 638f2f9bc67af472d318a6f0
paper_authors.avatarUrl: /avatars/f644218215d4a58c4de43486b96d94de.svg
paper_authors.fullname: Isma Hadji
paper_authors.type: user
paper_authors.user: hadjisma
paper_authors._id: 674d8ddeda397a3869dae2bd
paper_authors.name: Hai X. Pham
paper_authors._id: 674d8ddeda397a3869dae2be
paper_authors.name: Xiatian Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T22:08:02.924Z
paper_authors._id: 647ef9e7aa8c04bbf9362a8a
paper_authors.avatarUrl: /avatars/dc23e15acf7ee7f3b9f1a68c2716a66d.svg
paper_authors.fullname: Xiatian Zhu
paper_authors.type: user
paper_authors.user: Xiatian-Zhu
paper_authors._id: 674d8ddeda397a3869dae2bf
paper_authors.name: Georgios Tzimiropoulos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T22:07:56.398Z
paper_authors._id: 649db0dcd8f9b2e2cb47e4d7
paper_authors.avatarUrl: /avatars/e1fb64d5302271a3557ba306a85f2363.svg
paper_authors.fullname: Georgios Tzimiropoulos
paper_authors.type: user
paper_authors.user: tzimiro
paper_authors._id: 674d8ddeda397a3869dae2c0
paper_authors.name: Brais Martinez
paper_title: FAM Diffusion: Frequency and Attention Modulation for High-Resolution
    Image Generation with Stable Diffusion
paper_summary: Diffusion models are proficient at generating high-quality images. They are
  however effective only when operating at the resolution used during training.
  Inference at a scaled resolution leads to repetitive patterns and structural
  distortions. Retraining at higher resolutions quickly becomes prohibitive.
  Thus, methods enabling pre-existing diffusion models to operate at flexible
  test-time resolutions are highly desirable. Previous works suffer from frequent
  artifacts and often introduce large latency overheads. We propose two simple
  modules that combine to solve these issues. We introduce a Frequency Modulation
  (FM) module that leverages the Fourier domain to improve the global structure
  consistency, and an Attention Modulation (AM) module which improves the
  consistency of local texture patterns, a problem largely ignored in prior
  works. Our method, coined Fam diffusion, can seamlessly integrate into any
  latent diffusion model and requires no additional training. Extensive
  qualitative results highlight the effectiveness of our method in addressing
  structural and local artifacts, while quantitative results show
  state-of-the-art performance. Also, our method avoids redundant inference
  tricks for improved consistency such as patch-based or progressive generation,
  leading to negligible latency overheads.
paper_ai_summary: Fam diffusion integrates Frequency Modulation and Attention Modulation modules to enable high-quality image generation at flexible resolutions without additional training or significant latency.
paper_ai_keywords: Frequency Modulation Attention Modulation latent diffusion model