LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment
Recent advancements in text-to-video (T2V) generative models have shown
impressive capabilities. However, these models are still inadequate in aligning
synthesized videos with human preferences (e.g., accurately reflecting text
descriptions), which is particularly difficult to address, as human preferences
are inherently subjective and challenging to formalize as objective functions.
Therefore, this paper proposes LiFT, a novel fine-tuning method leveraging
human feedback for T2V model alignment. Specifically, we first construct a
Human Rating Annotation dataset, LiFT-HRA, consisting of approximately 10k
human annotations, each including a score and its corresponding rationale.
Based on this, we train a reward model LiFT-Critic to learn reward function
effectively, which serves as a proxy for human judgment, measuring the
alignment between given videos and human expectations. Lastly, we leverage the
learned reward function to align the T2V model by maximizing the
reward-weighted likelihood. As a case study, we apply our pipeline to
CogVideoX-2B, showing that the fine-tuned model outperforms the CogVideoX-5B
across all 16 metrics, highlighting the potential of human feedback in
improving the alignment and quality of synthesized videos.
67564cddee0958ae76fa740d
Yibin Wang
67564cddee0958ae76fa740e
Zhiyu Tan
67564cddee0958ae76fa740f
Junyan Wang
admin_assigned
2024-12-09T14:57:35.079Z
64dc6d60d8d7d6d6d9f3c78b
/avatars/b48a8760cd56d4b7b579f235be5637f3.svg
Junyan Wang
user
HotWater666
67564cddee0958ae76fa7410
Xiaomeng Yang
67564cddee0958ae76fa7411
Cheng Jin
67564cddee0958ae76fa7412
Hao Li
LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment
Recent advancements in text-to-video (T2V) generative models have shown
impressive capabilities. However, these models are still inadequate in aligning
synthesized videos with human preferences (e.g., accurately reflecting text
descriptions), which is particularly difficult to address, as human preferences
are inherently subjective and challenging to formalize as objective functions.
Therefore, this paper proposes LiFT, a novel fine-tuning method leveraging
human feedback for T2V model alignment. Specifically, we first construct a
Human Rating Annotation dataset, LiFT-HRA, consisting of approximately 10k
human annotations, each including a score and its corresponding rationale.
Based on this, we train a reward model LiFT-Critic to learn reward function
effectively, which serves as a proxy for human judgment, measuring the
alignment between given videos and human expectations. Lastly, we leverage the
learned reward function to align the T2V model by maximizing the
reward-weighted likelihood. As a case study, we apply our pipeline to
CogVideoX-2B, showing that the fine-tuned model outperforms the CogVideoX-5B
across all 16 metrics, highlighting the potential of human feedback in
improving the alignment and quality of synthesized videos.
LiFT fine-tunes text-to-video generative models using human feedback to improve video alignment with text descriptions, demonstrating superior performance over a larger model.
text-to-video fine-tuning human feedback dataset reward model reward function reward-weighted likelihood CogVideoX-2B CogVideoX-5B