Probabilistic Adaptation of Text-to-Video Models
Large text-to-video models trained on internet-scale data have demonstrated
exceptional capabilities in generating high-fidelity videos from arbitrary
textual descriptions. However, adapting these models to tasks with limited
domain-specific data, such as animation or robotics videos, poses a significant
computational challenge, since finetuning a pretrained large model can be
prohibitively expensive. Inspired by how a small modifiable component (e.g.,
prompts, prefix-tuning) can adapt a large language model to perform new tasks
without requiring access to the model weights, we investigate how to adapt a
large pretrained text-to-video model to a variety of downstream domains and
tasks without finetuning. In answering this question, we propose Video Adapter,
which leverages the score function of a large pretrained video diffusion model
as a probabilistic prior to guide the generation of a task-specific small video
model. Our experiments show that Video Adapter is capable of incorporating the
broad knowledge and preserving the high fidelity of a large pretrained video
model in a task-specific small video model that is able to generate
high-quality yet specialized videos on a variety of tasks such as animation,
egocentric modeling, and modeling of simulated and real-world robotics data.
More videos can be found on the website https://video-adapter.github.io/.
647ea166cfca67bc50ff87d4
Mengjiao Yang
647ea166cfca67bc50ff87d5
Yilun Du
extracted_confirmed
2023-06-06T03:21:53.895Z
63c9bd445fdc575773c732fe
/avatars/def472d1ab3fbf751225357c0932ae7e.svg
Yilun Du
user
yilundu
647ea166cfca67bc50ff87d6
Bo Dai
claimed_verified
2024-06-04T07:32:21.375Z
665e2bd0fbaa279db369b488
/avatars/8cbc18d17d2813617976dbe709235799.svg
Bo Dai
user
doubling
647ea166cfca67bc50ff87d7
Dale Schuurmans
647ea166cfca67bc50ff87d8
Joshua B. Tenenbaum
647ea166cfca67bc50ff87d9
Pieter Abbeel
Probabilistic Adaptation of Text-to-Video Models
Large text-to-video models trained on internet-scale data have demonstrated
exceptional capabilities in generating high-fidelity videos from arbitrary
textual descriptions. However, adapting these models to tasks with limited
domain-specific data, such as animation or robotics videos, poses a significant
computational challenge, since finetuning a pretrained large model can be
prohibitively expensive. Inspired by how a small modifiable component (e.g.,
prompts, prefix-tuning) can adapt a large language model to perform new tasks
without requiring access to the model weights, we investigate how to adapt a
large pretrained text-to-video model to a variety of downstream domains and
tasks without finetuning. In answering this question, we propose Video Adapter,
which leverages the score function of a large pretrained video diffusion model
as a probabilistic prior to guide the generation of a task-specific small video
model. Our experiments show that Video Adapter is capable of incorporating the
broad knowledge and preserving the high fidelity of a large pretrained video
model in a task-specific small video model that is able to generate
high-quality yet specialized videos on a variety of tasks such as animation,
egocentric modeling, and modeling of simulated and real-world robotics data.
More videos can be found on the website https://video-adapter.github.io/.
Video Adapter uses a large pretrained video diffusion model's score function to guide the generation of task-specific small video models without full finetuning.
text-to-video models diffusion models score function task-specific small video model animation egocentric modeling robotics