Mobile Video Diffusion
Video diffusion models have achieved impressive realism and controllability
but are limited by high computational demands, restricting their use on mobile
devices. This paper introduces the first mobile-optimized video diffusion
model. Starting from a spatio-temporal UNet from Stable Video Diffusion (SVD),
we reduce memory and computational cost by reducing the frame resolution,
incorporating multi-scale temporal representations, and introducing two novel
pruning schema to reduce the number of channels and temporal blocks.
Furthermore, we employ adversarial finetuning to reduce the denoising to a
single step. Our model, coined as MobileVD, is 523x more efficient (1817.2 vs.
4.34 TFLOPs) with a slight quality drop (FVD 149 vs. 171), generating latents
for a 14x512x256 px clip in 1.7 seconds on a Xiaomi-14 Pro. Our results are
available at https://qualcomm-ai-research.github.io/mobile-video-diffusion/
67598660eff1d97bd2cf2b7b
Haitam Ben Yahia
admin_assigned
2024-12-11T13:57:42.350Z
67598719eff1d97bd2cf6300
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/GoxSruke3IDDZP-vSyqyq.png
Haitam Ben Yahia/
user
haitamb
67598660eff1d97bd2cf2b7c
Denis Korzhenkov
67598660eff1d97bd2cf2b7d
Ioannis Lelekas
67598660eff1d97bd2cf2b7e
Amir Ghodrati
claimed_verified
2024-12-11T13:39:02.200Z
65d31d1bfe21569868432e96
/avatars/195207d241f4eab121e4fed6cf5d578d.svg
Amir
user
aghodrati
67598660eff1d97bd2cf2b7f
Amirhossein Habibian
claimed_verified
2024-12-11T13:39:04.250Z
628372aa09aa80237c6d2044
/avatars/c1e34dee217d73f5539a0807adc12c80.svg
Amir Habibian
user
habibian
Mobile Video Diffusion
Video diffusion models have achieved impressive realism and controllability
but are limited by high computational demands, restricting their use on mobile
devices. This paper introduces the first mobile-optimized video diffusion
model. Starting from a spatio-temporal UNet from Stable Video Diffusion (SVD),
we reduce memory and computational cost by reducing the frame resolution,
incorporating multi-scale temporal representations, and introducing two novel
pruning schema to reduce the number of channels and temporal blocks.
Furthermore, we employ adversarial finetuning to reduce the denoising to a
single step. Our model, coined as MobileVD, is 523x more efficient (1817.2 vs.
4.34 TFLOPs) with a slight quality drop (FVD 149 vs. 171), generating latents
for a 14x512x256 px clip in 1.7 seconds on a Xiaomi-14 Pro. Our results are
available at https://qualcomm-ai-research.github.io/mobile-video-diffusion/
A mobile-optimized video diffusion model reduces computational demands and maintains quality through pruning and adversarial finetuning.
spatio-temporal UNet MobileVD frame resolution multi-scale temporal representations pruning adversarial finetuning FVD