title: DenseDPO: Fine-Grained Temporal Preference Optimization for Video
    Diffusion Models
summary: Direct Preference Optimization (DPO) has recently been applied as a
  post-training technique for text-to-video diffusion models. To obtain training
  data, annotators are asked to provide preferences between two videos generated
  from independent noise. However, this approach prohibits fine-grained
  comparisons, and we point out that it biases the annotators towards low-motion
  clips as they often contain fewer visual artifacts. In this work, we introduce
  DenseDPO, a method that addresses these shortcomings by making three
  contributions. First, we create each video pair for DPO by denoising corrupted
  copies of a ground truth video. This results in aligned pairs with similar
  motion structures while differing in local details, effectively neutralizing
  the motion bias. Second, we leverage the resulting temporal alignment to label
  preferences on short segments rather than entire clips, yielding a denser and
  more precise learning signal. With only one-third of the labeled data, DenseDPO
  greatly improves motion generation over vanilla DPO, while matching it in text
  alignment, visual quality, and temporal consistency. Finally, we show that
  DenseDPO unlocks automatic preference annotation using off-the-shelf Vision
  Language Models (VLMs): GPT accurately predicts segment-level preferences
  similar to task-specifically fine-tuned video reward models, and DenseDPO
  trained on these labels achieves performance close to using human labels.
paper_authors._id: 68412f853c22997c7329f3a0
paper_authors.name: Ziyi Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T07:26:43.314Z
paper_authors._id: 62980664ff0acd7e027d6686
paper_authors.avatarUrl: /avatars/364d4c8432c24775a099641fc576dbdc.svg
paper_authors.fullname: Ziyi Wu
paper_authors.type: user
paper_authors.user: Dazitu616
paper_authors._id: 68412f853c22997c7329f3a1
paper_authors.name: Anil Kag
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:55:11.687Z
paper_authors._id: 66b01ee8e53bbad918362856
paper_authors.avatarUrl: /avatars/293529589a91dd7a95909d66727db224.svg
paper_authors.fullname: Anil Kag
paper_authors.type: user
paper_authors.user: anilkagak2
paper_authors._id: 68412f853c22997c7329f3a2
paper_authors.name: Ivan Skorokhodov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:55:18.188Z
paper_authors._id: 63610db13c7147fae7de88e3
paper_authors.avatarUrl: /avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
paper_authors.fullname: Ivan Skorokhodov
paper_authors.type: user
paper_authors.user: universome
paper_authors._id: 68412f853c22997c7329f3a3
paper_authors.name: Willi Menapace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:55:30.315Z
paper_authors._id: 6315358a362e3e95ea538081
paper_authors.avatarUrl: /avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
paper_authors.fullname: Willi Menapace
paper_authors.type: user
paper_authors.user: willi-menapace
paper_authors._id: 68412f853c22997c7329f3a4
paper_authors.name: Ashkan Mirzaei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:55:36.943Z
paper_authors._id: 634db15dd00bb5d92c3bd94f
paper_authors.avatarUrl: /avatars/9a6a1231bc5205911272d83527593f1a.svg
paper_authors.fullname: Ashkan Mirzaei
paper_authors.type: user
paper_authors.user: ashmrz
paper_authors._id: 68412f853c22997c7329f3a5
paper_authors.name: Igor Gilitschenski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:55:42.860Z
paper_authors._id: 64893f0b9ab28735f1a1ec0b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64893f0b9ab28735f1a1ec0b/2lPBjV16l2qLEWOKU2zwR.jpeg
paper_authors.fullname: Igor Gilitschenski
paper_authors.type: user
paper_authors.user: igilitschenski
paper_authors._id: 68412f853c22997c7329f3a6
paper_authors.name: Sergey Tulyakov
paper_authors._id: 68412f853c22997c7329f3a7
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:55:52.883Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_title: DenseDPO: Fine-Grained Temporal Preference Optimization for Video
    Diffusion Models
paper_summary: Direct Preference Optimization (DPO) has recently been applied as a
  post-training technique for text-to-video diffusion models. To obtain training
  data, annotators are asked to provide preferences between two videos generated
  from independent noise. However, this approach prohibits fine-grained
  comparisons, and we point out that it biases the annotators towards low-motion
  clips as they often contain fewer visual artifacts. In this work, we introduce
  DenseDPO, a method that addresses these shortcomings by making three
  contributions. First, we create each video pair for DPO by denoising corrupted
  copies of a ground truth video. This results in aligned pairs with similar
  motion structures while differing in local details, effectively neutralizing
  the motion bias. Second, we leverage the resulting temporal alignment to label
  preferences on short segments rather than entire clips, yielding a denser and
  more precise learning signal. With only one-third of the labeled data, DenseDPO
  greatly improves motion generation over vanilla DPO, while matching it in text
  alignment, visual quality, and temporal consistency. Finally, we show that
  DenseDPO unlocks automatic preference annotation using off-the-shelf Vision
  Language Models (VLMs): GPT accurately predicts segment-level preferences
  similar to task-specifically fine-tuned video reward models, and DenseDPO
  trained on these labels achieves performance close to using human labels.