title: T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model
    with Mixed Reward Feedback
summary: Diffusion-based text-to-video (T2V) models have achieved significant success
  but continue to be hampered by the slow sampling speed of their iterative
  sampling processes. To address the challenge, consistency models have been
  proposed to facilitate fast inference, albeit at the cost of sample quality. In
  this work, we aim to break the quality bottleneck of a video consistency model
  (VCM) to achieve both fast and high-quality video generation. We
  introduce T2V-Turbo, which integrates feedback from a mixture of differentiable
  reward models into the consistency distillation (CD) process of a pre-trained
  T2V model. Notably, we directly optimize rewards associated with single-step
  generations that arise naturally from computing the CD loss, effectively
  bypassing the memory constraints imposed by backpropagating gradients through
  an iterative sampling process. Remarkably, the 4-step generations from our
  T2V-Turbo achieve the highest total score on VBench, even surpassing Gen-2 and
  Pika. We further conduct human evaluations to corroborate the results,
  validating that the 4-step generations from our T2V-Turbo are preferred over
  the 50-step DDIM samples from their teacher models, representing more than a
  tenfold acceleration while improving video generation quality.
paper_authors._id: 6658893d24f3d17e4bdcbfd4
paper_authors.name: Jiachen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:25:30.471Z
paper_authors._id: 640d7ec5fdeaae13907fc488
paper_authors.avatarUrl: /avatars/acf43ea155105a51c8612dacc4725091.svg
paper_authors.fullname: Jiachen Li
paper_authors.type: user
paper_authors.user: jiachenli-ucsb
paper_authors._id: 6658893d24f3d17e4bdcbfd5
paper_authors.name: Weixi Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T07:30:21.568Z
paper_authors._id: 6259e9489326c60347dd2b6c
paper_authors.avatarUrl: /avatars/6aae6282ddee96b899b7384154e9b0a0.svg
paper_authors.fullname: Weixi Feng
paper_authors.type: user
paper_authors.user: weixifeng
paper_authors._id: 6658893d24f3d17e4bdcbfd6
paper_authors.name: Tsu-Jui Fu
paper_authors._id: 6658893d24f3d17e4bdcbfd7
paper_authors.name: Xinyi Wang
paper_authors._id: 6658893d24f3d17e4bdcbfd8
paper_authors.name: Sugato Basu
paper_authors._id: 6658893d24f3d17e4bdcbfd9
paper_authors.name: Wenhu Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-05-30T14:12:15.970Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_authors._id: 6658893d24f3d17e4bdcbfda
paper_authors.name: William Yang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:36:19.493Z
paper_authors._id: 6476607efb22e3b77f3f1452
paper_authors.avatarUrl: /avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
paper_authors.fullname: William Yang Wang
paper_authors.type: user
paper_authors.user: wangwilliamyang
paper_title: T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model
    with Mixed Reward Feedback
paper_summary: Diffusion-based text-to-video (T2V) models have achieved significant success
  but continue to be hampered by the slow sampling speed of their iterative
  sampling processes. To address the challenge, consistency models have been
  proposed to facilitate fast inference, albeit at the cost of sample quality. In
  this work, we aim to break the quality bottleneck of a video consistency model
  (VCM) to achieve both fast and high-quality video generation. We
  introduce T2V-Turbo, which integrates feedback from a mixture of differentiable
  reward models into the consistency distillation (CD) process of a pre-trained
  T2V model. Notably, we directly optimize rewards associated with single-step
  generations that arise naturally from computing the CD loss, effectively
  bypassing the memory constraints imposed by backpropagating gradients through
  an iterative sampling process. Remarkably, the 4-step generations from our
  T2V-Turbo achieve the highest total score on VBench, even surpassing Gen-2 and
  Pika. We further conduct human evaluations to corroborate the results,
  validating that the 4-step generations from our T2V-Turbo are preferred over
  the 50-step DDIM samples from their teacher models, representing more than a
  tenfold acceleration while improving video generation quality.
paper_ai_keywords: text-to-video (T2V) models diffusion-based models consistency models (VCM) Iterative sampling processes differentiable reward models consistency distillation (CD) single-step generations CD loss backpropagation VBench Gen-2 Pika 4-step generations DDIM samples video generation quality