SF-V: Single Forward Video Generation Model
Diffusion-based video generation models have demonstrated remarkable success
in obtaining high-fidelity videos through the iterative denoising process.
However, these models require multiple denoising steps during sampling,
resulting in high computational costs. In this work, we propose a novel
approach to obtain single-step video generation models by leveraging
adversarial training to fine-tune pre-trained video diffusion models. We show
that, through the adversarial training, the multi-steps video diffusion model,
i.e., Stable Video Diffusion (SVD), can be trained to perform single forward
pass to synthesize high-quality videos, capturing both temporal and spatial
dependencies in the video data. Extensive experiments demonstrate that our
method achieves competitive generation quality of synthesized videos with
significantly reduced computational overhead for the denoising process (i.e.,
around 23times speedup compared with SVD and 6times speedup compared with
existing works, with even better generation quality), paving the way for
real-time video synthesis and editing. More visualization results are made
publicly available at https://snap-research.github.io/SF-V.
666269c6fa2e0d45cd7c9414
Zhixing Zhang
claimed_verified
2024-06-07T09:20:12.160Z
63081b822d33960e6b78ccf0
/avatars/cbc3b007fdc050544a4bcc9881c63beb.svg
Zhixing Zhang
user
zhixingz
666269c6fa2e0d45cd7c9415
Yanyu Li
666269c6fa2e0d45cd7c9416
Yushu Wu
admin_assigned
2024-06-07T13:30:29.235Z
6566d82954c6bc01e54b65bf
/avatars/bcaaa2bcdb6ba8d4042a07def99cfa08.svg
Yushu Wu
user
wuyushuwys
666269c6fa2e0d45cd7c9417
Yanwu Xu
666269c6fa2e0d45cd7c9418
Anil Kag
666269c6fa2e0d45cd7c9419
Ivan Skorokhodov
admin_assigned
2024-06-07T13:31:04.054Z
63610db13c7147fae7de88e3
/avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
Ivan Skorokhodov
user
universome
666269c6fa2e0d45cd7c941a
Willi Menapace
admin_assigned
2024-06-07T13:31:09.708Z
6315358a362e3e95ea538081
/avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
Willi Menapace
user
willi-menapace
666269c6fa2e0d45cd7c941b
Aliaksandr Siarohin
admin_assigned
2024-06-07T13:31:16.337Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
666269c6fa2e0d45cd7c941c
Junli Cao
admin_assigned
2024-06-07T13:31:22.924Z
63f54aa73aa49d8cb97b84bc
/avatars/c73c5870039611ab9162daad46a1ba20.svg
junli cao
user
jlcao2
666269c6fa2e0d45cd7c941d
Dimitris Metaxas
666269c6fa2e0d45cd7c941e
Sergey Tulyakov
666269c6fa2e0d45cd7c941f
Jian Ren
claimed_verified
2024-06-07T09:20:03.858Z
61f19829233c91cbd2f79e70
/avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
Jian Ren
user
alanspike
SF-V: Single Forward Video Generation Model
Diffusion-based video generation models have demonstrated remarkable success
in obtaining high-fidelity videos through the iterative denoising process.
However, these models require multiple denoising steps during sampling,
resulting in high computational costs. In this work, we propose a novel
approach to obtain single-step video generation models by leveraging
adversarial training to fine-tune pre-trained video diffusion models. We show
that, through the adversarial training, the multi-steps video diffusion model,
i.e., Stable Video Diffusion (SVD), can be trained to perform single forward
pass to synthesize high-quality videos, capturing both temporal and spatial
dependencies in the video data. Extensive experiments demonstrate that our
method achieves competitive generation quality of synthesized videos with
significantly reduced computational overhead for the denoising process (i.e.,
around 23times speedup compared with SVD and 6times speedup compared with
existing works, with even better generation quality), paving the way for
real-time video synthesis and editing. More visualization results are made
publicly available at https://snap-research.github.io/SF-V.
Adversarial training transforms a multi-step diffusion video generation model into a single-step high-quality video synthesis model with reduced computational cost.
diffusion-based video generation iterative denoising adversarial training pre-trained video diffusion models single-step video generation Stable Video Diffusion temporal dependencies spatial dependencies competitive generation quality real-time video synthesis real-time video editing