MoViE: Mobile Diffusion for Video Editing
Recent progress in diffusion-based video editing has shown remarkable
potential for practical applications. However, these methods remain
prohibitively expensive and challenging to deploy on mobile devices. In this
study, we introduce a series of optimizations that render mobile video editing
feasible. Building upon the existing image editing model, we first optimize its
architecture and incorporate a lightweight autoencoder. Subsequently, we extend
classifier-free guidance distillation to multiple modalities, resulting in a
threefold on-device speedup. Finally, we reduce the number of sampling steps to
one by introducing a novel adversarial distillation scheme which preserves the
controllability of the editing process. Collectively, these optimizations
enable video editing at 12 frames per second on mobile devices, while
maintaining high quality. Our results are available at
https://qualcomm-ai-research.github.io/mobile-video-editing/
675985c7b51431a566ca75f9
Adil Karjauv
admin_assigned
2024-12-11T13:50:31.797Z
6759879b2f34dfa21f6789e8
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/CvoocygVY6KfegcQPpW0t.png
Adil Karjauv
user
mikolez
675985c7b51431a566ca75fa
Noor Fathima
admin_assigned
2024-12-11T13:50:37.583Z
64f066ac479ac4e6146908c1
/avatars/4dbd00c724039230334abb023f4b41a9.svg
Noor Fathima
user
Noor000
675985c7b51431a566ca75fb
Ioannis Lelekas
675985c7b51431a566ca75fc
Fatih Porikli
675985c7b51431a566ca75fd
Amir Ghodrati
claimed_verified
2024-12-11T13:39:05.718Z
65d31d1bfe21569868432e96
/avatars/195207d241f4eab121e4fed6cf5d578d.svg
Amir
user
aghodrati
675985c7b51431a566ca75fe
Amirhossein Habibian
claimed_verified
2024-12-11T13:39:07.789Z
628372aa09aa80237c6d2044
/avatars/c1e34dee217d73f5539a0807adc12c80.svg
Amir Habibian
user
habibian
MoViE: Mobile Diffusion for Video Editing
Recent progress in diffusion-based video editing has shown remarkable
potential for practical applications. However, these methods remain
prohibitively expensive and challenging to deploy on mobile devices. In this
study, we introduce a series of optimizations that render mobile video editing
feasible. Building upon the existing image editing model, we first optimize its
architecture and incorporate a lightweight autoencoder. Subsequently, we extend
classifier-free guidance distillation to multiple modalities, resulting in a
threefold on-device speedup. Finally, we reduce the number of sampling steps to
one by introducing a novel adversarial distillation scheme which preserves the
controllability of the editing process. Collectively, these optimizations
enable video editing at 12 frames per second on mobile devices, while
maintaining high quality. Our results are available at
https://qualcomm-ai-research.github.io/mobile-video-editing/
Optimizations including a lightweight autoencoder, classifier-free guidance distillation, and an adversarial distillation scheme enable high-speed, high-quality video editing on mobile devices.
diffusion-based video editing lightweight autoencoder classifier-free guidance distillation adversarial distillation mobile devices