title: StreamDiffusion: A Pipeline-level Solution for Real-time Interactive
    Generation
summary: We introduce StreamDiffusion, a real-time diffusion pipeline designed for
  interactive image generation. Existing diffusion models are adept at creating
  images from text or image prompts, yet they often fall short in real-time
  interaction. This limitation becomes particularly evident in scenarios
  involving continuous input, such as Metaverse, live video streaming, and
  broadcasting, where high throughput is imperative. To address this, we present
  a novel approach that transforms the original sequential denoising into the
  batching denoising process. Stream Batch eliminates the conventional
  wait-and-interact approach and enables fluid and high throughput streams. To
  handle the frequency disparity between data input and model throughput, we
  design a novel input-output queue for parallelizing the streaming process.
  Moreover, the existing diffusion pipeline uses classifier-free guidance(CFG),
  which requires additional U-Net computation. To mitigate the redundant
  computations, we propose a novel residual classifier-free guidance (RCFG)
  algorithm that reduces the number of negative conditional denoising steps to
  only one or even zero. Besides, we introduce a stochastic similarity
  filter(SSF) to optimize power consumption. Our Stream Batch achieves around
  1.5x speedup compared to the sequential denoising method at different denoising
  levels. The proposed RCFG leads to speeds up to 2.05x higher than the
  conventional CFG. Combining the proposed strategies and existing mature
  acceleration tools makes the image-to-image generation achieve up-to 91.07fps
  on one RTX4090, improving the throughputs of AutoPipline developed by Diffusers
  over 59.56x. Furthermore, our proposed StreamDiffusion also significantly
  reduces the energy consumption by 2.39x on one RTX3060 and 1.99x on one
  RTX4090, respectively.
paper_authors._id: 6583b6021fb17c3a9aad3dbf
paper_authors.name: Akio Kodaira
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-21T07:44:50.797Z
paper_authors._id: 63fedca388b9695964c33ad8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63fedca388b9695964c33ad8/XFE8Tm9yRRbo6XdDw7c2O.png
paper_authors.fullname: Aki
paper_authors.type: user
paper_authors.user: AkiCumulo
paper_authors._id: 6583b6021fb17c3a9aad3dc0
paper_authors.name: Chenfeng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:33:11.637Z
paper_authors._id: 654ca71d5255ee86711b52c5
paper_authors.avatarUrl: /avatars/52bf00fd74c8db5643c4daa185c678e6.svg
paper_authors.fullname: Chenfeng Xu
paper_authors.type: user
paper_authors.user: chenfengx
paper_authors._id: 6583b6021fb17c3a9aad3dc1
paper_authors.name: Toshiki Hazama
paper_authors._id: 6583b6021fb17c3a9aad3dc2
paper_authors.name: Takanori Yoshimoto
paper_authors._id: 6583b6021fb17c3a9aad3dc3
paper_authors.name: Kohei Ohno
paper_authors._id: 6583b6021fb17c3a9aad3dc4
paper_authors.name: Shogo Mitsuhori
paper_authors._id: 6583b6021fb17c3a9aad3dc5
paper_authors.name: Soichi Sugano
paper_authors._id: 6583b6021fb17c3a9aad3dc6
paper_authors.name: Hanying Cho
paper_authors._id: 6583b6021fb17c3a9aad3dc7
paper_authors.name: Zhijian Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:33:57.469Z
paper_authors._id: 650dac79b959b0e1d41d7378
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
paper_authors.fullname: Zhijian Liu
paper_authors.type: user
paper_authors.user: zhijianliu
paper_authors._id: 6583b6021fb17c3a9aad3dc8
paper_authors.name: Kurt Keutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:34:23.279Z
paper_authors._id: 6251bf4b183aa4266924ad91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
paper_authors.fullname: Kurt Keutzer
paper_authors.type: user
paper_authors.user: kurtkeutzer
paper_title: StreamDiffusion: A Pipeline-level Solution for Real-time Interactive
    Generation
paper_summary: We introduce StreamDiffusion, a real-time diffusion pipeline designed for
  interactive image generation. Existing diffusion models are adept at creating
  images from text or image prompts, yet they often fall short in real-time
  interaction. This limitation becomes particularly evident in scenarios
  involving continuous input, such as Metaverse, live video streaming, and
  broadcasting, where high throughput is imperative. To address this, we present
  a novel approach that transforms the original sequential denoising into the
  batching denoising process. Stream Batch eliminates the conventional
  wait-and-interact approach and enables fluid and high throughput streams. To
  handle the frequency disparity between data input and model throughput, we
  design a novel input-output queue for parallelizing the streaming process.
  Moreover, the existing diffusion pipeline uses classifier-free guidance(CFG),
  which requires additional U-Net computation. To mitigate the redundant
  computations, we propose a novel residual classifier-free guidance (RCFG)
  algorithm that reduces the number of negative conditional denoising steps to
  only one or even zero. Besides, we introduce a stochastic similarity
  filter(SSF) to optimize power consumption. Our Stream Batch achieves around
  1.5x speedup compared to the sequential denoising method at different denoising
  levels. The proposed RCFG leads to speeds up to 2.05x higher than the
  conventional CFG. Combining the proposed strategies and existing mature
  acceleration tools makes the image-to-image generation achieve up-to 91.07fps
  on one RTX4090, improving the throughputs of AutoPipline developed by Diffusers
  over 59.56x. Furthermore, our proposed StreamDiffusion also significantly
  reduces the energy consumption by 2.39x on one RTX3060 and 1.99x on one
  RTX4090, respectively.
paper_ai_summary: StreamDiffusion, a real-time diffusion pipeline, enhances interactive image generation by implementing batching denoising, residual classifier-free guidance, and stochastic similarity filtering, which boost throughput and reduce energy consumption.
paper_ai_keywords: StreamDiffusion diffusion pipeline interactive image generation streaming denoising batch denoising Stream Batch classifier-free guidance CFG residual classifier-free guidance RCFG stochastic similarity filter SSF throughput energy consumption RTX4090 RTX3060 AutoPipline Diffusers