DistriFusion: Distributed Parallel Inference for High-Resolution
  Diffusion Models
Diffusion models have achieved great success in synthesizing high-quality
images. However, generating high-resolution images with diffusion models is
still challenging due to the enormous computational costs, resulting in a
prohibitive latency for interactive applications. In this paper, we propose
DistriFusion to tackle this problem by leveraging parallelism across multiple
GPUs. Our method splits the model input into multiple patches and assigns each
patch to a GPU. However, na\"{\i}vely implementing such an algorithm breaks the
interaction between patches and loses fidelity, while incorporating such an
interaction will incur tremendous communication overhead. To overcome this
dilemma, we observe the high similarity between the input from adjacent
diffusion steps and propose displaced patch parallelism, which takes advantage
of the sequential nature of the diffusion process by reusing the pre-computed
feature maps from the previous timestep to provide context for the current
step. Therefore, our method supports asynchronous communication, which can be
pipelined by computation. Extensive experiments show that our method can be
applied to recent Stable Diffusion XL with no quality degradation and achieve
up to a 6.1times speedup on eight NVIDIA A100s compared to one. Our code is
publicly available at https://github.com/mit-han-lab/distrifuser.
65e16b31c9fd4a6beced4d71
Muyang Li
admin_assigned
2024-03-01T12:55:00.727Z
63129589bbaa385279d1826e
https://cdn-avatars.huggingface.co/v1/production/uploads/63129589bbaa385279d1826e/0AUs3q4ngRZ-wXuY1jP9G.jpeg
Muyang Li
user
Lmxyy
65e16b31c9fd4a6beced4d72
Tianle Cai
admin_assigned
2024-03-01T12:55:11.069Z
64459357c525660aa20be337
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1fryBXRuSvAgMIDkpWBpI.jpeg
Tianle Cai
user
tianlecai
65e16b31c9fd4a6beced4d73
Jiaxin Cao
admin_assigned
2024-03-01T12:55:23.689Z
648bfdcd026d7dba2b74ad90
/avatars/e4541e314a1701a9887f63626365ab0a.svg
Jiaxin Cao
user
caojx
65e16b31c9fd4a6beced4d74
Qinsheng Zhang
admin_assigned
2024-03-01T12:55:33.417Z
62fb1886d8a067d3b36ffbdf
/avatars/539ccdaa4d3f94124d589669eff7c787.svg
zhang
user
qinsheng
65e16b31c9fd4a6beced4d75
Han Cai
claimed_verified
2024-10-28T11:29:14.161Z
650e2b14c945dfc9386a7e28
/avatars/0a9be20aa53f4c52a2d1a8b02d4093ea.svg
Han Cai
user
han-cai
65e16b31c9fd4a6beced4d76
Junjie Bai
admin_assigned
2024-03-01T12:56:02.263Z
659466c489145cbc7c713d70
/avatars/19d7518500089d75205c37ba747511b6.svg
Junjie Bai
user
OasisBai
65e16b31c9fd4a6beced4d77
Yangqing Jia
admin_assigned
2024-03-01T12:56:09.277Z
628f2248b90dde28ef563fbd
https://cdn-avatars.huggingface.co/v1/production/uploads/628f2248b90dde28ef563fbd/j_y8PlW9OC0E9_PzwTBSt.jpeg
Yangqing Jia
user
Yangqing
65e16b31c9fd4a6beced4d78
Ming-Yu Liu
admin_assigned
2024-03-01T12:56:24.809Z
62f049afdf4b93aad5c7f2d6
/avatars/e272e58ad996733d7098e50248e5b57e.svg
Ming-Yu Liu
user
mingyuliutw
65e16b31c9fd4a6beced4d79
Kai Li
65e16b31c9fd4a6beced4d7a
Song Han
DistriFusion: Distributed Parallel Inference for High-Resolution
  Diffusion Models
Diffusion models have achieved great success in synthesizing high-quality
images. However, generating high-resolution images with diffusion models is
still challenging due to the enormous computational costs, resulting in a
prohibitive latency for interactive applications. In this paper, we propose
DistriFusion to tackle this problem by leveraging parallelism across multiple
GPUs. Our method splits the model input into multiple patches and assigns each
patch to a GPU. However, na\"{\i}vely implementing such an algorithm breaks the
interaction between patches and loses fidelity, while incorporating such an
interaction will incur tremendous communication overhead. To overcome this
dilemma, we observe the high similarity between the input from adjacent
diffusion steps and propose displaced patch parallelism, which takes advantage
of the sequential nature of the diffusion process by reusing the pre-computed
feature maps from the previous timestep to provide context for the current
step. Therefore, our method supports asynchronous communication, which can be
pipelined by computation. Extensive experiments show that our method can be
applied to recent Stable Diffusion XL with no quality degradation and achieve
up to a 6.1times speedup on eight NVIDIA A100s compared to one. Our code is
publicly available at https://github.com/mit-han-lab/distrifuser.
DistriFusion leverages parallelism and displaced patch parallelism to significantly speed up diffusion models like Stable Diffusion XL without compromising image quality.
diffusion models displaced patch parallelism feature maps parallelism GPU Stable Diffusion XL asynchronous communication