title: Efficient Video Diffusion Models via Content-Frame Motion-Latent
    Decomposition
summary: Video diffusion models have recently made great progress in generation
  quality, but are still limited by the high memory and computational
  requirements. This is because current video diffusion models often attempt to
  process high-dimensional videos directly. To tackle this issue, we propose
  content-motion latent diffusion model (CMD), a novel efficient extension of
  pretrained image diffusion models for video generation. Specifically, we
  propose an autoencoder that succinctly encodes a video as a combination of a
  content frame (like an image) and a low-dimensional motion latent
  representation. The former represents the common content, and the latter
  represents the underlying motion in the video, respectively. We generate the
  content frame by fine-tuning a pretrained image diffusion model, and we
  generate the motion latent representation by training a new lightweight
  diffusion model. A key innovation here is the design of a compact latent space
  that can directly utilizes a pretrained image diffusion model, which has not
  been done in previous latent video diffusion models. This leads to considerably
  better quality generation and reduced computational costs. For instance, CMD
  can sample a video 7.7times faster than prior approaches by generating a
  video of 512times1024 resolution and length 16 in 3.1 seconds. Moreover, CMD
  achieves an FVD score of 212.7 on WebVid-10M, 27.3% better than the previous
  state-of-the-art of 292.4.
paper_authors._id: 65fcea5d9904f33f4af62fe9
paper_authors.name: Sihyun Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T10:14:30.371Z
paper_authors._id: 64722d14609ae9f5636b6e7d
paper_authors.avatarUrl: /avatars/153447cd0b1c354a2692817a39aec5b7.svg
paper_authors.fullname: Sihyun Yu
paper_authors.type: user
paper_authors.user: sihyun-yu
paper_authors._id: 65fcea5d9904f33f4af62fea
paper_authors.name: Weili Nie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T10:14:36.504Z
paper_authors._id: 64c1a69e226e016da8450ae2
paper_authors.avatarUrl: /avatars/54c161e8b8543244ed13cbe47017624e.svg
paper_authors.fullname: Weili Nie
paper_authors.type: user
paper_authors.user: xiaoli08
paper_authors._id: 65fcea5d9904f33f4af62feb
paper_authors.name: De-An Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T10:14:44.631Z
paper_authors._id: 641d1c5ec3983aa94915c162
paper_authors.avatarUrl: /avatars/127985b837ecf61e43c835deee578b5e.svg
paper_authors.fullname: De-An Huang
paper_authors.type: user
paper_authors.user: deahuang
paper_authors._id: 65fcea5d9904f33f4af62fec
paper_authors.name: Boyi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T10:15:35.813Z
paper_authors._id: 620dd3888528f797e88cb9b5
paper_authors.avatarUrl: /avatars/af04728788d78fe7d6375e19e32a535e.svg
paper_authors.fullname: Boyi Li
paper_authors.type: user
paper_authors.user: Boyiliee
paper_authors._id: 65fcea5d9904f33f4af62fed
paper_authors.name: Jinwoo Shin
paper_authors._id: 65fcea5d9904f33f4af62fee
paper_authors.name: Anima Anandkumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T10:15:48.249Z
paper_authors._id: 6532920b3e385cfc6002938d
paper_authors.avatarUrl: /avatars/cb9cc6d2733031582c83f56dc6cd1dd5.svg
paper_authors.fullname: Anima Anandkumar
paper_authors.type: user
paper_authors.user: animakumar
paper_title: Efficient Video Diffusion Models via Content-Frame Motion-Latent
    Decomposition
paper_summary: Video diffusion models have recently made great progress in generation
  quality, but are still limited by the high memory and computational
  requirements. This is because current video diffusion models often attempt to
  process high-dimensional videos directly. To tackle this issue, we propose
  content-motion latent diffusion model (CMD), a novel efficient extension of
  pretrained image diffusion models for video generation. Specifically, we
  propose an autoencoder that succinctly encodes a video as a combination of a
  content frame (like an image) and a low-dimensional motion latent
  representation. The former represents the common content, and the latter
  represents the underlying motion in the video, respectively. We generate the
  content frame by fine-tuning a pretrained image diffusion model, and we
  generate the motion latent representation by training a new lightweight
  diffusion model. A key innovation here is the design of a compact latent space
  that can directly utilizes a pretrained image diffusion model, which has not
  been done in previous latent video diffusion models. This leads to considerably
  better quality generation and reduced computational costs. For instance, CMD
  can sample a video 7.7times faster than prior approaches by generating a
  video of 512times1024 resolution and length 16 in 3.1 seconds. Moreover, CMD
  achieves an FVD score of 212.7 on WebVid-10M, 27.3% better than the previous
  state-of-the-art of 292.4.
paper_ai_summary: A new content-motion latent diffusion model (CMD) enhances video generation quality and efficiency by decoupling content and motion and leveraging pretrained image diffusion models.
paper_ai_keywords: video diffusion models autoencoder content frame low-dimensional motion representation pretrained image diffusion model compact latent space FVD score WebVid-10M