title: AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising
summary: Diffusion models have garnered significant interest from the community for
  their great generative ability across various applications. However, their
  typical multi-step sequential-denoising nature gives rise to high cumulative
  latency, thereby precluding the possibilities of parallel computation. To
  address this, we introduce AsyncDiff, a universal and plug-and-play
  acceleration scheme that enables model parallelism across multiple devices. Our
  approach divides the cumbersome noise prediction model into multiple
  components, assigning each to a different device. To break the dependency chain
  between these components, it transforms the conventional sequential denoising
  into an asynchronous process by exploiting the high similarity between hidden
  states in consecutive diffusion steps. Consequently, each component is
  facilitated to compute in parallel on separate devices. The proposed strategy
  significantly reduces inference latency while minimally impacting the
  generative quality. Specifically, for the Stable Diffusion v2.1, AsyncDiff
  achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with
  only a slight reduction of 0.38 in CLIP Score, on four NVIDIA A5000 GPUs. Our
  experiments also demonstrate that AsyncDiff can be readily applied to video
  diffusion models with encouraging performances. The code is available at
  https://github.com/czg1225/AsyncDiff.
paper_authors._id: 666915f90a19d52d441b7730
paper_authors.name: Zigeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T09:31:30.267Z
paper_authors._id: 65811eeaa2284a018e51f1ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/dH8UZj6Kk5HJkI1DItCNm.jpeg
paper_authors.fullname: Zigeng Chen
paper_authors.type: user
paper_authors.user: Zigeng
paper_authors._id: 666915f90a19d52d441b7731
paper_authors.name: Xinyin Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T09:31:36.686Z
paper_authors._id: 64396ebc21221ac7411852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64396ebc21221ac7411852b3/SR0dC8N0bdj9tZFxYPpSf.jpeg
paper_authors.fullname: Xinyin Ma
paper_authors.type: user
paper_authors.user: horseee
paper_authors._id: 666915f90a19d52d441b7732
paper_authors.name: Gongfan Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T09:31:43.353Z
paper_authors._id: 646a1939c37ca1e12308fe81
paper_authors.avatarUrl: /avatars/752e9d86018e7d33ad8bcd741203fd86.svg
paper_authors.fullname: Gongfan Fang
paper_authors.type: user
paper_authors.user: Vinnnf
paper_authors._id: 666915f90a19d52d441b7733
paper_authors.name: Zhenxiong Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T09:31:52.919Z
paper_authors._id: 664ef197d3f8759dddfbc765
paper_authors.avatarUrl: /avatars/fd3f8a844b391be6c1ac2ea89b228f5a.svg
paper_authors.fullname: Tan Zhen Xiong
paper_authors.type: user
paper_authors.user: ZhenXiong
paper_authors._id: 666915f90a19d52d441b7734
paper_authors.name: Xinchao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T09:32:05.522Z
paper_authors._id: 63fc03a50aab060792ffef39
paper_authors.avatarUrl: /avatars/9d5b1bb2a41928e08176b703935133ab.svg
paper_authors.fullname: Wangxinchao
paper_authors.type: user
paper_authors.user: wxcTest
paper_title: AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising
paper_summary: Diffusion models have garnered significant interest from the community for
  their great generative ability across various applications. However, their
  typical multi-step sequential-denoising nature gives rise to high cumulative
  latency, thereby precluding the possibilities of parallel computation. To
  address this, we introduce AsyncDiff, a universal and plug-and-play
  acceleration scheme that enables model parallelism across multiple devices. Our
  approach divides the cumbersome noise prediction model into multiple
  components, assigning each to a different device. To break the dependency chain
  between these components, it transforms the conventional sequential denoising
  into an asynchronous process by exploiting the high similarity between hidden
  states in consecutive diffusion steps. Consequently, each component is
  facilitated to compute in parallel on separate devices. The proposed strategy
  significantly reduces inference latency while minimally impacting the
  generative quality. Specifically, for the Stable Diffusion v2.1, AsyncDiff
  achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with
  only a slight reduction of 0.38 in CLIP Score, on four NVIDIA A5000 GPUs. Our
  experiments also demonstrate that AsyncDiff can be readily applied to video
  diffusion models with encouraging performances. The code is available at
  https://github.com/czg1225/AsyncDiff.
paper_ai_summary: AsyncDiff, a parallel computation scheme for diffusion models, reduces inference latency without significantly impacting generative quality, demonstrating speedups on image and video tasks.
paper_ai_keywords: diffusion models sequential denoising model parallelism noise prediction model asynchronous process hidden states Stable Diffusion CLIP Score video diffusion models