title: Improving Video Generation with Human Feedback
summary: Video generation has achieved significant advances through rectified flow
  techniques, but issues like unsmooth motion and misalignment between videos and
  prompts persist. In this work, we develop a systematic pipeline that harnesses
  human feedback to mitigate these problems and refine the video generation
  model. Specifically, we begin by constructing a large-scale human preference
  dataset focused on modern video generation models, incorporating pairwise
  annotations across multi-dimensions. We then introduce VideoReward, a
  multi-dimensional video reward model, and examine how annotations and various
  design choices impact its rewarding efficacy. From a unified reinforcement
  learning perspective aimed at maximizing reward with KL regularization, we
  introduce three alignment algorithms for flow-based models by extending those
  from diffusion models. These include two training-time strategies: direct
  preference optimization for flow (Flow-DPO) and reward weighted regression for
  flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies
  reward guidance directly to noisy videos. Experimental results indicate that
  VideoReward significantly outperforms existing reward models, and Flow-DPO
  demonstrates superior performance compared to both Flow-RWR and standard
  supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom
  weights to multiple objectives during inference, meeting personalized video
  quality needs. Project page: https://gongyeliu.github.io/videoalign.
paper_authors._id: 679319848d46289f90266168
paper_authors.name: Jie Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-24T09:07:53.235Z
paper_authors._id: 639be86b59473c6ae02ef9c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639be86b59473c6ae02ef9c4/gw34RBCVZCOkcAA79xUr3.png
paper_authors.fullname: Jie Liu
paper_authors.type: user
paper_authors.user: jieliu
paper_authors._id: 679319848d46289f90266169
paper_authors.name: Gongye Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T13:37:43.546Z
paper_authors._id: 6553316bf151de82f6a23e1d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6553316bf151de82f6a23e1d/GTBkSj4Fa3OoyM6Muz_Sc.jpeg
paper_authors.fullname: Gongye Liu
paper_authors.type: user
paper_authors.user: liuhuohuo
paper_authors._id: 679319848d46289f9026616a
paper_authors.name: Jiajun Liang
paper_authors._id: 679319848d46289f9026616b
paper_authors.name: Ziyang Yuan
paper_authors._id: 679319848d46289f9026616c
paper_authors.name: Xiaokun Liu
paper_authors._id: 679319848d46289f9026616d
paper_authors.name: Mingwu Zheng
paper_authors._id: 679319848d46289f9026616e
paper_authors.name: Xiele Wu
paper_authors._id: 679319848d46289f9026616f
paper_authors.name: Qiulin Wang
paper_authors._id: 679319848d46289f90266170
paper_authors.name: Wenyu Qin
paper_authors._id: 679319848d46289f90266171
paper_authors.name: Menghan Xia
paper_authors._id: 679319848d46289f90266172
paper_authors.name: Xintao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-24T09:07:51.248Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 679319848d46289f90266173
paper_authors.name: Xiaohong Liu
paper_authors._id: 679319848d46289f90266174
paper_authors.name: Fei Yang
paper_authors._id: 679319848d46289f90266175
paper_authors.name: Pengfei Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-17T14:42:51.336Z
paper_authors._id: 662f93942510ef5735d7ad00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
paper_authors.fullname: magicwpf
paper_authors.type: user
paper_authors.user: magicwpf
paper_authors._id: 679319848d46289f90266176
paper_authors.name: Di Zhang
paper_authors._id: 679319848d46289f90266177
paper_authors.name: Kun Gai
paper_authors._id: 679319848d46289f90266178
paper_authors.name: Yujiu Yang
paper_authors._id: 679319848d46289f90266179
paper_authors.name: Wanli Ouyang
paper_title: Improving Video Generation with Human Feedback
paper_summary: Video generation has achieved significant advances through rectified flow
  techniques, but issues like unsmooth motion and misalignment between videos and
  prompts persist. In this work, we develop a systematic pipeline that harnesses
  human feedback to mitigate these problems and refine the video generation
  model. Specifically, we begin by constructing a large-scale human preference
  dataset focused on modern video generation models, incorporating pairwise
  annotations across multi-dimensions. We then introduce VideoReward, a
  multi-dimensional video reward model, and examine how annotations and various
  design choices impact its rewarding efficacy. From a unified reinforcement
  learning perspective aimed at maximizing reward with KL regularization, we
  introduce three alignment algorithms for flow-based models by extending those
  from diffusion models. These include two training-time strategies: direct
  preference optimization for flow (Flow-DPO) and reward weighted regression for
  flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies
  reward guidance directly to noisy videos. Experimental results indicate that
  VideoReward significantly outperforms existing reward models, and Flow-DPO
  demonstrates superior performance compared to both Flow-RWR and standard
  supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom
  weights to multiple objectives during inference, meeting personalized video
  quality needs. Project page: https://gongyeliu.github.io/videoalign.
paper_ai_summary: VideoReward and alignment algorithms improve video generation by addressing unsmooth motion and misalignment using human feedback in a unified reinforcement learning framework.
paper_ai_keywords: rectified flow techniques human preference dataset VideoReward multi-dimensional video reward model reinforcement learning KL regularization flow-based models diffusion models direct preference optimization reward weighted regression reward guidance Flow-DPO Flow-RWR Flow-NRG standard supervised fine-tuning