T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model
  with Mixed Reward Feedback
Diffusion-based text-to-video (T2V) models have achieved significant success
but continue to be hampered by the slow sampling speed of their iterative
sampling processes. To address the challenge, consistency models have been
proposed to facilitate fast inference, albeit at the cost of sample quality. In
this work, we aim to break the quality bottleneck of a video consistency model
(VCM) to achieve both fast and high-quality video generation. We
introduce T2V-Turbo, which integrates feedback from a mixture of differentiable
reward models into the consistency distillation (CD) process of a pre-trained
T2V model. Notably, we directly optimize rewards associated with single-step
generations that arise naturally from computing the CD loss, effectively
bypassing the memory constraints imposed by backpropagating gradients through
an iterative sampling process. Remarkably, the 4-step generations from our
T2V-Turbo achieve the highest total score on VBench, even surpassing Gen-2 and
Pika. We further conduct human evaluations to corroborate the results,
validating that the 4-step generations from our T2V-Turbo are preferred over
the 50-step DDIM samples from their teacher models, representing more than a
tenfold acceleration while improving video generation quality.
6658893d24f3d17e4bdcbfd4
Jiachen Li
claimed_verified
2024-05-31T06:25:30.471Z
640d7ec5fdeaae13907fc488
/avatars/acf43ea155105a51c8612dacc4725091.svg
Jiachen Li
user
jiachenli-ucsb
6658893d24f3d17e4bdcbfd5
Weixi Feng
claimed_verified
2024-06-12T07:30:21.568Z
6259e9489326c60347dd2b6c
/avatars/6aae6282ddee96b899b7384154e9b0a0.svg
Weixi Feng
user
weixifeng
6658893d24f3d17e4bdcbfd6
Tsu-Jui Fu
6658893d24f3d17e4bdcbfd7
Xinyi Wang
6658893d24f3d17e4bdcbfd8
Sugato Basu
6658893d24f3d17e4bdcbfd9
Wenhu Chen
extracted_pending
2024-05-30T14:12:15.970Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
6658893d24f3d17e4bdcbfda
William Yang Wang
admin_assigned
2024-05-31T10:36:19.493Z
6476607efb22e3b77f3f1452
/avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
William Yang Wang
user
wangwilliamyang
T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model
  with Mixed Reward Feedback
Diffusion-based text-to-video (T2V) models have achieved significant success
but continue to be hampered by the slow sampling speed of their iterative
sampling processes. To address the challenge, consistency models have been
proposed to facilitate fast inference, albeit at the cost of sample quality. In
this work, we aim to break the quality bottleneck of a video consistency model
(VCM) to achieve both fast and high-quality video generation. We
introduce T2V-Turbo, which integrates feedback from a mixture of differentiable
reward models into the consistency distillation (CD) process of a pre-trained
T2V model. Notably, we directly optimize rewards associated with single-step
generations that arise naturally from computing the CD loss, effectively
bypassing the memory constraints imposed by backpropagating gradients through
an iterative sampling process. Remarkably, the 4-step generations from our
T2V-Turbo achieve the highest total score on VBench, even surpassing Gen-2 and
Pika. We further conduct human evaluations to corroborate the results,
validating that the 4-step generations from our T2V-Turbo are preferred over
the 50-step DDIM samples from their teacher models, representing more than a
tenfold acceleration while improving video generation quality.
T2V-Turbo improves video generation quality and speed by integrating differentiable reward models into consistency distillation, achieving superior results to models with iterative sampling.
diffusion-based text-to-video iterative sampling processes consistency models video consistency model T2V-Turbo differentiable reward models consistency distillation VBench DDIM