Inflation with Diffusion: Efficient Temporal Adaptation for
  Text-to-Video Super-Resolution
We propose an efficient diffusion-based text-to-video super-resolution (SR)
tuning approach that leverages the readily learned capacity of pixel level
image diffusion model to capture spatial information for video generation. To
accomplish this goal, we design an efficient architecture by inflating the
weightings of the text-to-image SR model into our video generation framework.
Additionally, we incorporate a temporal adapter to ensure temporal coherence
across video frames. We investigate different tuning approaches based on our
inflated architecture and report trade-offs between computational costs and
super-resolution quality. Empirical evaluation, both quantitative and
qualitative, on the Shutterstock video dataset, demonstrates that our approach
is able to perform text-to-video SR generation with good visual quality and
temporal consistency. To evaluate temporal coherence, we also present
visualizations in video format in
https://drive.google.com/drive/folders/1YVc-KMSJqOrEUdQWVaI-Yfu8Vsfu_1aO?usp=sharing .
65adcf8a356bf23b4a954769
Xin Yuan
claimed_verified
2024-01-22T21:11:27.773Z
65aeb94cda1c18bd6dbd3319
/avatars/c01cd557875715861fab2a0d9b390b4d.svg
Xin Yuan
user
yuanxzzz
65adcf8a356bf23b4a95476a
Jinoo Baek
claimed_verified
2024-01-24T08:09:37.994Z
636623a9aa6a4af607370a47
/avatars/ae2f377e7d85b22ec244f90c8434b87f.svg
Jinwoo Baek
user
jinoo
65adcf8a356bf23b4a95476b
Keyang Xu
65adcf8a356bf23b4a95476c
Omer Tov
claimed_verified
2024-01-22T14:51:45.262Z
623889093657dc674c30954e
/avatars/a5a193d231e01448cc78ed5bba52bff5.svg
Omer Tov
user
omertov
65adcf8a356bf23b4a95476d
Hongliang Fei
claimed_verified
2024-01-22T21:11:30.504Z
65aeb55e66ad44e2d567620c
/avatars/36b98e143c465b852a26f14307e571ca.svg
Hongliang Fei
user
feihongliang0
Inflation with Diffusion: Efficient Temporal Adaptation for
  Text-to-Video Super-Resolution
We propose an efficient diffusion-based text-to-video super-resolution (SR)
tuning approach that leverages the readily learned capacity of pixel level
image diffusion model to capture spatial information for video generation. To
accomplish this goal, we design an efficient architecture by inflating the
weightings of the text-to-image SR model into our video generation framework.
Additionally, we incorporate a temporal adapter to ensure temporal coherence
across video frames. We investigate different tuning approaches based on our
inflated architecture and report trade-offs between computational costs and
super-resolution quality. Empirical evaluation, both quantitative and
qualitative, on the Shutterstock video dataset, demonstrates that our approach
is able to perform text-to-video SR generation with good visual quality and
temporal consistency. To evaluate temporal coherence, we also present
visualizations in video format in
https://drive.google.com/drive/folders/1YVc-KMSJqOrEUdQWVaI-Yfu8Vsfu_1aO?usp=sharing .
An efficient diffusion-based method inflates a text-to-image super-resolution framework to generate high-quality, temporally coherent text-to-video super-resolution content.
diffusion-based text-to-video super-resolution pixel level image diffusion model inflated architecture temporal adapter temporal coherence Shutterstock video dataset