title: Probabilistic Adaptation of Text-to-Video Models
summary: Large text-to-video models trained on internet-scale data have demonstrated
  exceptional capabilities in generating high-fidelity videos from arbitrary
  textual descriptions. However, adapting these models to tasks with limited
  domain-specific data, such as animation or robotics videos, poses a significant
  computational challenge, since finetuning a pretrained large model can be
  prohibitively expensive. Inspired by how a small modifiable component (e.g.,
  prompts, prefix-tuning) can adapt a large language model to perform new tasks
  without requiring access to the model weights, we investigate how to adapt a
  large pretrained text-to-video model to a variety of downstream domains and
  tasks without finetuning. In answering this question, we propose Video Adapter,
  which leverages the score function of a large pretrained video diffusion model
  as a probabilistic prior to guide the generation of a task-specific small video
  model. Our experiments show that Video Adapter is capable of incorporating the
  broad knowledge and preserving the high fidelity of a large pretrained video
  model in a task-specific small video model that is able to generate
  high-quality yet specialized videos on a variety of tasks such as animation,
  egocentric modeling, and modeling of simulated and real-world robotics data.
  More videos can be found on the website https://video-adapter.github.io/.
paper_authors._id: 647ea166cfca67bc50ff87d4
paper_authors.name: Mengjiao Yang
paper_authors._id: 647ea166cfca67bc50ff87d5
paper_authors.name: Yilun Du
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-06T03:21:53.895Z
paper_authors._id: 63c9bd445fdc575773c732fe
paper_authors.avatarUrl: /avatars/def472d1ab3fbf751225357c0932ae7e.svg
paper_authors.fullname: Yilun Du
paper_authors.type: user
paper_authors.user: yilundu
paper_authors._id: 647ea166cfca67bc50ff87d6
paper_authors.name: Bo Dai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-04T07:32:21.375Z
paper_authors._id: 665e2bd0fbaa279db369b488
paper_authors.avatarUrl: /avatars/8cbc18d17d2813617976dbe709235799.svg
paper_authors.fullname: Bo Dai
paper_authors.type: user
paper_authors.user: doubling
paper_authors._id: 647ea166cfca67bc50ff87d7
paper_authors.name: Dale Schuurmans
paper_authors._id: 647ea166cfca67bc50ff87d8
paper_authors.name: Joshua B. Tenenbaum
paper_authors._id: 647ea166cfca67bc50ff87d9
paper_authors.name: Pieter Abbeel
paper_title: Probabilistic Adaptation of Text-to-Video Models
paper_summary: Large text-to-video models trained on internet-scale data have demonstrated
  exceptional capabilities in generating high-fidelity videos from arbitrary
  textual descriptions. However, adapting these models to tasks with limited
  domain-specific data, such as animation or robotics videos, poses a significant
  computational challenge, since finetuning a pretrained large model can be
  prohibitively expensive. Inspired by how a small modifiable component (e.g.,
  prompts, prefix-tuning) can adapt a large language model to perform new tasks
  without requiring access to the model weights, we investigate how to adapt a
  large pretrained text-to-video model to a variety of downstream domains and
  tasks without finetuning. In answering this question, we propose Video Adapter,
  which leverages the score function of a large pretrained video diffusion model
  as a probabilistic prior to guide the generation of a task-specific small video
  model. Our experiments show that Video Adapter is capable of incorporating the
  broad knowledge and preserving the high fidelity of a large pretrained video
  model in a task-specific small video model that is able to generate
  high-quality yet specialized videos on a variety of tasks such as animation,
  egocentric modeling, and modeling of simulated and real-world robotics data.
  More videos can be found on the website https://video-adapter.github.io/.
paper_ai_keywords: score function video diffusion model probabilistic prior Video Adapter high-fidelity videos textual descriptions domain-specific data animation egocentric modeling real-world robotics data simulated robotics data