Efficient Video Diffusion Models via Content-Frame Motion-Latent
  Decomposition
Video diffusion models have recently made great progress in generation
quality, but are still limited by the high memory and computational
requirements. This is because current video diffusion models often attempt to
process high-dimensional videos directly. To tackle this issue, we propose
content-motion latent diffusion model (CMD), a novel efficient extension of
pretrained image diffusion models for video generation. Specifically, we
propose an autoencoder that succinctly encodes a video as a combination of a
content frame (like an image) and a low-dimensional motion latent
representation. The former represents the common content, and the latter
represents the underlying motion in the video, respectively. We generate the
content frame by fine-tuning a pretrained image diffusion model, and we
generate the motion latent representation by training a new lightweight
diffusion model. A key innovation here is the design of a compact latent space
that can directly utilizes a pretrained image diffusion model, which has not
been done in previous latent video diffusion models. This leads to considerably
better quality generation and reduced computational costs. For instance, CMD
can sample a video 7.7times faster than prior approaches by generating a
video of 512times1024 resolution and length 16 in 3.1 seconds. Moreover, CMD
achieves an FVD score of 212.7 on WebVid-10M, 27.3% better than the previous
state-of-the-art of 292.4.
65fcea5d9904f33f4af62fe9
Sihyun Yu
admin_assigned
2024-03-22T10:14:30.371Z
64722d14609ae9f5636b6e7d
/avatars/153447cd0b1c354a2692817a39aec5b7.svg
Sihyun Yu
user
sihyun-yu
65fcea5d9904f33f4af62fea
Weili Nie
admin_assigned
2024-03-22T10:14:36.504Z
64c1a69e226e016da8450ae2
/avatars/54c161e8b8543244ed13cbe47017624e.svg
Weili Nie
user
xiaoli08
65fcea5d9904f33f4af62feb
De-An Huang
admin_assigned
2024-03-22T10:14:44.631Z
641d1c5ec3983aa94915c162
/avatars/127985b837ecf61e43c835deee578b5e.svg
De-An Huang
user
deahuang
65fcea5d9904f33f4af62fec
Boyi Li
admin_assigned
2024-03-22T10:15:35.813Z
620dd3888528f797e88cb9b5
/avatars/af04728788d78fe7d6375e19e32a535e.svg
Boyi Li
user
Boyiliee
65fcea5d9904f33f4af62fed
Jinwoo Shin
65fcea5d9904f33f4af62fee
Anima Anandkumar
admin_assigned
2024-03-22T10:15:48.249Z
6532920b3e385cfc6002938d
/avatars/cb9cc6d2733031582c83f56dc6cd1dd5.svg
Anima Anandkumar
user
animakumar
Efficient Video Diffusion Models via Content-Frame Motion-Latent
  Decomposition
Video diffusion models have recently made great progress in generation
quality, but are still limited by the high memory and computational
requirements. This is because current video diffusion models often attempt to
process high-dimensional videos directly. To tackle this issue, we propose
content-motion latent diffusion model (CMD), a novel efficient extension of
pretrained image diffusion models for video generation. Specifically, we
propose an autoencoder that succinctly encodes a video as a combination of a
content frame (like an image) and a low-dimensional motion latent
representation. The former represents the common content, and the latter
represents the underlying motion in the video, respectively. We generate the
content frame by fine-tuning a pretrained image diffusion model, and we
generate the motion latent representation by training a new lightweight
diffusion model. A key innovation here is the design of a compact latent space
that can directly utilizes a pretrained image diffusion model, which has not
been done in previous latent video diffusion models. This leads to considerably
better quality generation and reduced computational costs. For instance, CMD
can sample a video 7.7times faster than prior approaches by generating a
video of 512times1024 resolution and length 16 in 3.1 seconds. Moreover, CMD
achieves an FVD score of 212.7 on WebVid-10M, 27.3% better than the previous
state-of-the-art of 292.4.
A new content-motion latent diffusion model (CMD) enhances video generation quality and efficiency by decoupling content and motion and leveraging pretrained image diffusion models.
video diffusion models autoencoder content frame low-dimensional motion representation pretrained image diffusion model compact latent space FVD score WebVid-10M