title: OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video
    Diffusion Model
summary: Variational Autoencoder (VAE), compressing videos into latent
  representations, is a crucial preceding component of Latent Video Diffusion
  Models (LVDMs). With the same reconstruction quality, the more sufficient the
  VAE's compression for videos is, the more efficient the LVDMs are. However,
  most LVDMs utilize 2D image VAE, whose compression for videos is only in the
  spatial dimension and often ignored in the temporal dimension. How to conduct
  temporal compression for videos in a VAE to obtain more concise latent
  representations while promising accurate reconstruction is seldom explored. To
  fill this gap, we propose an omni-dimension compression VAE, named OD-VAE,
  which can temporally and spatially compress videos. Although OD-VAE's more
  sufficient compression brings a great challenge to video reconstruction, it can
  still achieve high reconstructed accuracy by our fine design. To obtain a
  better trade-off between video reconstruction quality and compression speed,
  four variants of OD-VAE are introduced and analyzed. In addition, a novel tail
  initialization is designed to train OD-VAE more efficiently, and a novel
  inference strategy is proposed to enable OD-VAE to handle videos of arbitrary
  length with limited GPU memory. Comprehensive experiments on video
  reconstruction and LVDM-based video generation demonstrate the effectiveness
  and efficiency of our proposed methods.
paper_authors._id: 66d7f0aa6776d11459adae38
paper_authors.name: Liuhan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:38:55.516Z
paper_authors._id: 65c0588569429d85dc594a85
paper_authors.avatarUrl: /avatars/822203d982519b999247a53235e30c39.svg
paper_authors.fullname: liuhan
paper_authors.type: user
paper_authors.user: LiuhanChen
paper_authors._id: 66d7f0aa6776d11459adae39
paper_authors.name: Zongjian Li
paper_authors._id: 66d7f0aa6776d11459adae3a
paper_authors.name: Bin Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:16:54.034Z
paper_authors._id: 6367a8175bb06007ea099b8f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
paper_authors.fullname: linbin
paper_authors.type: user
paper_authors.user: LanguageBind
paper_authors._id: 66d7f0aa6776d11459adae3b
paper_authors.name: Bin Zhu
paper_authors._id: 66d7f0aa6776d11459adae3c
paper_authors.name: Qian Wang
paper_authors._id: 66d7f0aa6776d11459adae3d
paper_authors.name: Shenghai Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-06T09:06:08.240Z
paper_authors._id: 63468720dd6d90d82ccf3450
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
paper_authors.fullname: YSH
paper_authors.type: user
paper_authors.user: BestWishYsh
paper_authors._id: 66d7f0aa6776d11459adae3e
paper_authors.name: Xing Zhou
paper_authors._id: 66d7f0aa6776d11459adae3f
paper_authors.name: Xinghua Cheng
paper_authors._id: 66d7f0aa6776d11459adae40
paper_authors.name: Li Yuan
paper_title: OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video
    Diffusion Model
paper_summary: Variational Autoencoder (VAE), compressing videos into latent
  representations, is a crucial preceding component of Latent Video Diffusion
  Models (LVDMs). With the same reconstruction quality, the more sufficient the
  VAE's compression for videos is, the more efficient the LVDMs are. However,
  most LVDMs utilize 2D image VAE, whose compression for videos is only in the
  spatial dimension and often ignored in the temporal dimension. How to conduct
  temporal compression for videos in a VAE to obtain more concise latent
  representations while promising accurate reconstruction is seldom explored. To
  fill this gap, we propose an omni-dimension compression VAE, named OD-VAE,
  which can temporally and spatially compress videos. Although OD-VAE's more
  sufficient compression brings a great challenge to video reconstruction, it can
  still achieve high reconstructed accuracy by our fine design. To obtain a
  better trade-off between video reconstruction quality and compression speed,
  four variants of OD-VAE are introduced and analyzed. In addition, a novel tail
  initialization is designed to train OD-VAE more efficiently, and a novel
  inference strategy is proposed to enable OD-VAE to handle videos of arbitrary
  length with limited GPU memory. Comprehensive experiments on video
  reconstruction and LVDM-based video generation demonstrate the effectiveness
  and efficiency of our proposed methods.
paper_ai_keywords: Variational Autoencoder (VAE) Latent Video Diffusion Models (LVDMs) omni-dimension compression VAE (OD-VAE) adaptive-dimension compression VAE (ADD-VAE) tail initialization inference strategy latent trade-off between video reconstruction quality and compression speed high reconstructed accuracy dual-point compression precision dimension compression temporal compression spatial compression latent representations video reconstruction compressing videos GPU days Variational Autoencoder (VAE) Latent Video Diffusion Models (LVDMs) omni-dimension compression VAE (OD-VAE) dual-point compression temporal compression spatial