StreamDiffusion: A Pipeline-level Solution for Real-time Interactive
  Generation
We introduce StreamDiffusion, a real-time diffusion pipeline designed for
interactive image generation. Existing diffusion models are adept at creating
images from text or image prompts, yet they often fall short in real-time
interaction. This limitation becomes particularly evident in scenarios
involving continuous input, such as Metaverse, live video streaming, and
broadcasting, where high throughput is imperative. To address this, we present
a novel approach that transforms the original sequential denoising into the
batching denoising process. Stream Batch eliminates the conventional
wait-and-interact approach and enables fluid and high throughput streams. To
handle the frequency disparity between data input and model throughput, we
design a novel input-output queue for parallelizing the streaming process.
Moreover, the existing diffusion pipeline uses classifier-free guidance(CFG),
which requires additional U-Net computation. To mitigate the redundant
computations, we propose a novel residual classifier-free guidance (RCFG)
algorithm that reduces the number of negative conditional denoising steps to
only one or even zero. Besides, we introduce a stochastic similarity
filter(SSF) to optimize power consumption. Our Stream Batch achieves around
1.5x speedup compared to the sequential denoising method at different denoising
levels. The proposed RCFG leads to speeds up to 2.05x higher than the
conventional CFG. Combining the proposed strategies and existing mature
acceleration tools makes the image-to-image generation achieve up-to 91.07fps
on one RTX4090, improving the throughputs of AutoPipline developed by Diffusers
over 59.56x. Furthermore, our proposed StreamDiffusion also significantly
reduces the energy consumption by 2.39x on one RTX3060 and 1.99x on one
RTX4090, respectively.
6583b6021fb17c3a9aad3dbf
Akio Kodaira
claimed_verified
2023-12-21T07:44:50.797Z
63fedca388b9695964c33ad8
https://cdn-avatars.huggingface.co/v1/production/uploads/63fedca388b9695964c33ad8/XFE8Tm9yRRbo6XdDw7c2O.png
Aki
user
AkiCumulo
6583b6021fb17c3a9aad3dc0
Chenfeng Xu
admin_assigned
2023-12-21T08:33:11.637Z
654ca71d5255ee86711b52c5
/avatars/52bf00fd74c8db5643c4daa185c678e6.svg
Chenfeng Xu
user
chenfengx
6583b6021fb17c3a9aad3dc1
Toshiki Hazama
6583b6021fb17c3a9aad3dc2
Takanori Yoshimoto
6583b6021fb17c3a9aad3dc3
Kohei Ohno
6583b6021fb17c3a9aad3dc4
Shogo Mitsuhori
6583b6021fb17c3a9aad3dc5
Soichi Sugano
6583b6021fb17c3a9aad3dc6
Hanying Cho
6583b6021fb17c3a9aad3dc7
Zhijian Liu
admin_assigned
2023-12-21T08:33:57.469Z
650dac79b959b0e1d41d7378
https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
Zhijian Liu
user
zhijianliu
6583b6021fb17c3a9aad3dc8
Kurt Keutzer
admin_assigned
2023-12-21T08:34:23.279Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
StreamDiffusion: A Pipeline-level Solution for Real-time Interactive
  Generation
We introduce StreamDiffusion, a real-time diffusion pipeline designed for
interactive image generation. Existing diffusion models are adept at creating
images from text or image prompts, yet they often fall short in real-time
interaction. This limitation becomes particularly evident in scenarios
involving continuous input, such as Metaverse, live video streaming, and
broadcasting, where high throughput is imperative. To address this, we present
a novel approach that transforms the original sequential denoising into the
batching denoising process. Stream Batch eliminates the conventional
wait-and-interact approach and enables fluid and high throughput streams. To
handle the frequency disparity between data input and model throughput, we
design a novel input-output queue for parallelizing the streaming process.
Moreover, the existing diffusion pipeline uses classifier-free guidance(CFG),
which requires additional U-Net computation. To mitigate the redundant
computations, we propose a novel residual classifier-free guidance (RCFG)
algorithm that reduces the number of negative conditional denoising steps to
only one or even zero. Besides, we introduce a stochastic similarity
filter(SSF) to optimize power consumption. Our Stream Batch achieves around
1.5x speedup compared to the sequential denoising method at different denoising
levels. The proposed RCFG leads to speeds up to 2.05x higher than the
conventional CFG. Combining the proposed strategies and existing mature
acceleration tools makes the image-to-image generation achieve up-to 91.07fps
on one RTX4090, improving the throughputs of AutoPipline developed by Diffusers
over 59.56x. Furthermore, our proposed StreamDiffusion also significantly
reduces the energy consumption by 2.39x on one RTX3060 and 1.99x on one
RTX4090, respectively.
StreamDiffusion, a real-time diffusion pipeline, enhances interactive image generation by implementing batching denoising, residual classifier-free guidance, and stochastic similarity filtering, which boost throughput and reduce energy consumption.
StreamDiffusion diffusion pipeline interactive image generation streaming denoising batch denoising Stream Batch classifier-free guidance CFG residual classifier-free guidance RCFG stochastic similarity filter SSF throughput energy consumption RTX4090 RTX3060 AutoPipline Diffusers