T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through
  Data, Reward, and Conditional Guidance Design
In this paper, we focus on enhancing a diffusion-based text-to-video (T2V)
model during the post-training phase by distilling a highly capable consistency
model from a pretrained T2V model. Our proposed method, T2V-Turbo-v2,
introduces a significant advancement by integrating various supervision
signals, including high-quality training data, reward model feedback, and
conditional guidance, into the consistency distillation process. Through
comprehensive ablation studies, we highlight the crucial importance of
tailoring datasets to specific learning objectives and the effectiveness of
learning from diverse reward models for enhancing both the visual quality and
text-video alignment. Additionally, we highlight the vast design space of
conditional guidance strategies, which centers on designing an effective energy
function to augment the teacher ODE solver. We demonstrate the potential of
this approach by extracting motion guidance from the training datasets and
incorporating it into the ODE solver, showcasing its effectiveness in improving
the motion quality of the generated videos with the improved motion-related
metrics from VBench and T2V-CompBench. Empirically, our T2V-Turbo-v2
establishes a new state-of-the-art result on VBench, with a Total score of
85.13, surpassing proprietary systems such as Gen-3 and Kling.
6707527a2f420b4b07e4da90
Jiachen Li
claimed_verified
2024-10-10T08:00:33.880Z
640d7ec5fdeaae13907fc488
/avatars/acf43ea155105a51c8612dacc4725091.svg
Jiachen Li
user
jiachenli-ucsb
6707527a2f420b4b07e4da91
Qian Long
6707527a2f420b4b07e4da92
Jian Zheng
claimed_verified
2024-10-11T07:44:09.821Z
6424d5b196f39ebc04165381
/avatars/35c95b1c1449921189585b1606b32db7.svg
Zheng
user
JulyApril
6707527a2f420b4b07e4da93
Xiaofeng Gao
admin_assigned
2024-10-10T09:35:42.663Z
64485435e21484883406f5fd
/avatars/910b920a6a6e8c7e0c7f7406554dd55b.svg
Xiaofeng Gao
user
xfgao
6707527a2f420b4b07e4da94
Robinson Piramuthu
admin_assigned
2024-10-10T09:35:37.667Z
639a65cdabfa5e2fccf18b82
/avatars/945f6478d7a219174ffb1ccd14d7bad5.svg
Robinson Piramuthu
user
rpiramuthu
6707527a2f420b4b07e4da95
Wenhu Chen
extracted_pending
2024-10-10T04:05:18.584Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
6707527a2f420b4b07e4da96
William Yang Wang
admin_assigned
2024-10-10T09:38:43.138Z
6476607efb22e3b77f3f1452
/avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
William Yang Wang
user
wangwilliamyang
T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through
  Data, Reward, and Conditional Guidance Design
In this paper, we focus on enhancing a diffusion-based text-to-video (T2V)
model during the post-training phase by distilling a highly capable consistency
model from a pretrained T2V model. Our proposed method, T2V-Turbo-v2,
introduces a significant advancement by integrating various supervision
signals, including high-quality training data, reward model feedback, and
conditional guidance, into the consistency distillation process. Through
comprehensive ablation studies, we highlight the crucial importance of
tailoring datasets to specific learning objectives and the effectiveness of
learning from diverse reward models for enhancing both the visual quality and
text-video alignment. Additionally, we highlight the vast design space of
conditional guidance strategies, which centers on designing an effective energy
function to augment the teacher ODE solver. We demonstrate the potential of
this approach by extracting motion guidance from the training datasets and
incorporating it into the ODE solver, showcasing its effectiveness in improving
the motion quality of the generated videos with the improved motion-related
metrics from VBench and T2V-CompBench. Empirically, our T2V-Turbo-v2
establishes a new state-of-the-art result on VBench, with a Total score of
85.13, surpassing proprietary systems such as Gen-3 and Kling.
T2V-Turbo-v2 enhances a diffusion-based text-to-video model through consistency distillation with tailored dataset supervision, reward model feedback, and conditional guidance, setting a new state-of-the-art on VBench.
diffusion-based text-to-video T2V-Turbo-v2 consistency model high-quality training data reward model feedback conditional guidance energy function ODE solver VBench T2V-CompBench motion guidance