title: DistriFusion: Distributed Parallel Inference for High-Resolution
    Diffusion Models
summary: Diffusion models have achieved great success in synthesizing high-quality
  images. However, generating high-resolution images with diffusion models is
  still challenging due to the enormous computational costs, resulting in a
  prohibitive latency for interactive applications. In this paper, we propose
  DistriFusion to tackle this problem by leveraging parallelism across multiple
  GPUs. Our method splits the model input into multiple patches and assigns each
  patch to a GPU. However, na\"{\i}vely implementing such an algorithm breaks the
  interaction between patches and loses fidelity, while incorporating such an
  interaction will incur tremendous communication overhead. To overcome this
  dilemma, we observe the high similarity between the input from adjacent
  diffusion steps and propose displaced patch parallelism, which takes advantage
  of the sequential nature of the diffusion process by reusing the pre-computed
  feature maps from the previous timestep to provide context for the current
  step. Therefore, our method supports asynchronous communication, which can be
  pipelined by computation. Extensive experiments show that our method can be
  applied to recent Stable Diffusion XL with no quality degradation and achieve
  up to a 6.1times speedup on eight NVIDIA A100s compared to one. Our code is
  publicly available at https://github.com/mit-han-lab/distrifuser.
paper_authors._id: 65e16b31c9fd4a6beced4d71
paper_authors.name: Muyang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T12:55:00.727Z
paper_authors._id: 63129589bbaa385279d1826e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63129589bbaa385279d1826e/0AUs3q4ngRZ-wXuY1jP9G.jpeg
paper_authors.fullname: Muyang Li
paper_authors.type: user
paper_authors.user: Lmxyy
paper_authors._id: 65e16b31c9fd4a6beced4d72
paper_authors.name: Tianle Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T12:55:11.069Z
paper_authors._id: 64459357c525660aa20be337
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1fryBXRuSvAgMIDkpWBpI.jpeg
paper_authors.fullname: Tianle Cai
paper_authors.type: user
paper_authors.user: tianlecai
paper_authors._id: 65e16b31c9fd4a6beced4d73
paper_authors.name: Jiaxin Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T12:55:23.689Z
paper_authors._id: 648bfdcd026d7dba2b74ad90
paper_authors.avatarUrl: /avatars/e4541e314a1701a9887f63626365ab0a.svg
paper_authors.fullname: Jiaxin Cao
paper_authors.type: user
paper_authors.user: caojx
paper_authors._id: 65e16b31c9fd4a6beced4d74
paper_authors.name: Qinsheng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T12:55:33.417Z
paper_authors._id: 62fb1886d8a067d3b36ffbdf
paper_authors.avatarUrl: /avatars/539ccdaa4d3f94124d589669eff7c787.svg
paper_authors.fullname: zhang
paper_authors.type: user
paper_authors.user: qinsheng
paper_authors._id: 65e16b31c9fd4a6beced4d75
paper_authors.name: Han Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:29:14.161Z
paper_authors._id: 650e2b14c945dfc9386a7e28
paper_authors.avatarUrl: /avatars/0a9be20aa53f4c52a2d1a8b02d4093ea.svg
paper_authors.fullname: Han Cai
paper_authors.type: user
paper_authors.user: han-cai
paper_authors._id: 65e16b31c9fd4a6beced4d76
paper_authors.name: Junjie Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T12:56:02.263Z
paper_authors._id: 659466c489145cbc7c713d70
paper_authors.avatarUrl: /avatars/19d7518500089d75205c37ba747511b6.svg
paper_authors.fullname: Junjie Bai
paper_authors.type: user
paper_authors.user: OasisBai
paper_authors._id: 65e16b31c9fd4a6beced4d77
paper_authors.name: Yangqing Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T12:56:09.277Z
paper_authors._id: 628f2248b90dde28ef563fbd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628f2248b90dde28ef563fbd/j_y8PlW9OC0E9_PzwTBSt.jpeg
paper_authors.fullname: Yangqing Jia
paper_authors.type: user
paper_authors.user: Yangqing
paper_authors._id: 65e16b31c9fd4a6beced4d78
paper_authors.name: Ming-Yu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T12:56:24.809Z
paper_authors._id: 62f049afdf4b93aad5c7f2d6
paper_authors.avatarUrl: /avatars/e272e58ad996733d7098e50248e5b57e.svg
paper_authors.fullname: Ming-Yu Liu
paper_authors.type: user
paper_authors.user: mingyuliutw
paper_authors._id: 65e16b31c9fd4a6beced4d79
paper_authors.name: Kai Li
paper_authors._id: 65e16b31c9fd4a6beced4d7a
paper_authors.name: Song Han
paper_title: DistriFusion: Distributed Parallel Inference for High-Resolution
    Diffusion Models
paper_summary: Diffusion models have achieved great success in synthesizing high-quality
  images. However, generating high-resolution images with diffusion models is
  still challenging due to the enormous computational costs, resulting in a
  prohibitive latency for interactive applications. In this paper, we propose
  DistriFusion to tackle this problem by leveraging parallelism across multiple
  GPUs. Our method splits the model input into multiple patches and assigns each
  patch to a GPU. However, na\"{\i}vely implementing such an algorithm breaks the
  interaction between patches and loses fidelity, while incorporating such an
  interaction will incur tremendous communication overhead. To overcome this
  dilemma, we observe the high similarity between the input from adjacent
  diffusion steps and propose displaced patch parallelism, which takes advantage
  of the sequential nature of the diffusion process by reusing the pre-computed
  feature maps from the previous timestep to provide context for the current
  step. Therefore, our method supports asynchronous communication, which can be
  pipelined by computation. Extensive experiments show that our method can be
  applied to recent Stable Diffusion XL with no quality degradation and achieve
  up to a 6.1times speedup on eight NVIDIA A100s compared to one. Our code is
  publicly available at https://github.com/mit-han-lab/distrifuser.
paper_ai_summary: DistriFusion leverages parallelism and displaced patch parallelism to significantly speed up diffusion models like Stable Diffusion XL without compromising image quality.
paper_ai_keywords: diffusion models displaced patch parallelism feature maps parallelism GPU Stable Diffusion XL asynchronous communication