title: Video Diffusion Alignment via Reward Gradients
summary: We have made significant progress towards building foundational video
  diffusion models. As these models are trained using large-scale unsupervised
  data, it has become crucial to adapt these models to specific downstream tasks.
  Adapting these models via supervised fine-tuning requires collecting target
  datasets of videos, which is challenging and tedious. In this work, we utilize
  pre-trained reward models that are learned via preferences on top of powerful
  vision discriminative models to adapt video diffusion models. These models
  contain dense gradient information with respect to generated RGB pixels, which
  is critical to efficient learning in complex search spaces, such as videos. We
  show that backpropagating gradients from these reward models to a video
  diffusion model can allow for compute and sample efficient alignment of the
  video diffusion model. We show results across a variety of reward models and
  video diffusion models, demonstrating that our approach can learn much more
  efficiently in terms of reward queries and computation than prior gradient-free
  approaches. Our code, model weights,and more visualization are available at
  https://vader-vid.github.io.
paper_authors._id: 6690952268b20cd8e62a7ca4
paper_authors.name: Mihir Prabhudesai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T08:57:06.020Z
paper_authors._id: 6310ff7dd43c55e811f8772f
paper_authors.avatarUrl: /avatars/0dcab81771329d13cc62b08e55a71c9a.svg
paper_authors.fullname: Mihir Prabhudesai
paper_authors.type: user
paper_authors.user: mihirpd
paper_authors._id: 6690952268b20cd8e62a7ca5
paper_authors.name: Russell Mendonca
paper_authors._id: 6690952268b20cd8e62a7ca6
paper_authors.name: Zheyang Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T09:25:47.447Z
paper_authors._id: 65fe6f3a3d247974ba711d47
paper_authors.avatarUrl: /avatars/c4f78f0cde2803458f4ab397e95ef652.svg
paper_authors.fullname: Zheyang Qin
paper_authors.type: user
paper_authors.user: zheyangqin
paper_authors._id: 6690952268b20cd8e62a7ca7
paper_authors.name: Katerina Fragkiadaki
paper_authors._id: 6690952268b20cd8e62a7ca8
paper_authors.name: Deepak Pathak
paper_title: Video Diffusion Alignment via Reward Gradients
paper_summary: We have made significant progress towards building foundational video
  diffusion models. As these models are trained using large-scale unsupervised
  data, it has become crucial to adapt these models to specific downstream tasks.
  Adapting these models via supervised fine-tuning requires collecting target
  datasets of videos, which is challenging and tedious. In this work, we utilize
  pre-trained reward models that are learned via preferences on top of powerful
  vision discriminative models to adapt video diffusion models. These models
  contain dense gradient information with respect to generated RGB pixels, which
  is critical to efficient learning in complex search spaces, such as videos. We
  show that backpropagating gradients from these reward models to a video
  diffusion model can allow for compute and sample efficient alignment of the
  video diffusion model. We show results across a variety of reward models and
  video diffusion models, demonstrating that our approach can learn much more
  efficiently in terms of reward queries and computation than prior gradient-free
  approaches. Our code, model weights,and more visualization are available at
  https://vader-vid.github.io.
paper_ai_summary: Utilizing pre-trained reward models to adapt video diffusion models with gradient-based feedback enhances efficiency and performance compared to gradient-free methods.
paper_ai_keywords: video diffusion models pre-trained reward models vision discriminative models backpropagating gradients gradient information RGB pixels efficient learning reward queries computational efficiency