title: Large Motion Video Autoencoding with Cross-modal Video VAE
summary: Learning a robust video Variational Autoencoder (VAE) is essential for
  reducing video redundancy and facilitating efficient video generation. Directly
  applying image VAEs to individual frames in isolation can result in temporal
  inconsistencies and suboptimal compression rates due to a lack of temporal
  compression. Existing Video VAEs have begun to address temporal compression;
  however, they often suffer from inadequate reconstruction performance. In this
  paper, we present a novel and powerful video autoencoder capable of
  high-fidelity video encoding. First, we observe that entangling spatial and
  temporal compression by merely extending the image VAE to a 3D VAE can
  introduce motion blur and detail distortion artifacts. Thus, we propose
  temporal-aware spatial compression to better encode and decode the spatial
  information. Additionally, we integrate a lightweight motion compression model
  for further temporal compression. Second, we propose to leverage the textual
  information inherent in text-to-video datasets and incorporate text guidance
  into our model. This significantly enhances reconstruction quality,
  particularly in terms of detail preservation and temporal stability. Third, we
  further improve the versatility of our model through joint training on both
  images and videos, which not only enhances reconstruction quality but also
  enables the model to perform both image and video autoencoding. Extensive
  evaluations against strong recent baselines demonstrate the superior
  performance of our method. The project website can be found
  at~https://yzxing87.github.io/vae/{https://yzxing87.github.io/vae/}.
paper_authors._id: 676a5396207235e4b020dfda
paper_authors.name: Yazhou Xing
paper_authors._id: 676a5396207235e4b020dfdb
paper_authors.name: Yang Fei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:39:56.502Z
paper_authors._id: 648ffe95669191ccb6772a2e
paper_authors.avatarUrl: /avatars/025cac7dc64ea9ef2074754c92086baa.svg
paper_authors.fullname: Yang Fei
paper_authors.type: user
paper_authors.user: sunfly
paper_authors._id: 676a5396207235e4b020dfdc
paper_authors.name: Yingqing He
paper_authors._id: 676a5396207235e4b020dfdd
paper_authors.name: Jingye Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:05:38.574Z
paper_authors._id: 6478a982256b62e219917d67
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/PUJ-N2cQxgEmDGfyjajyA.jpeg
paper_authors.fullname: JingyeChen22
paper_authors.type: user
paper_authors.user: JingyeChen22
paper_authors._id: 676a5396207235e4b020dfde
paper_authors.name: Jiaxin Xie
paper_authors._id: 676a5396207235e4b020dfdf
paper_authors.name: Xiaowei Chi
paper_authors._id: 676a5396207235e4b020dfe0
paper_authors.name: Qifeng Chen
paper_title: Large Motion Video Autoencoding with Cross-modal Video VAE
paper_summary: Learning a robust video Variational Autoencoder (VAE) is essential for
  reducing video redundancy and facilitating efficient video generation. Directly
  applying image VAEs to individual frames in isolation can result in temporal
  inconsistencies and suboptimal compression rates due to a lack of temporal
  compression. Existing Video VAEs have begun to address temporal compression;
  however, they often suffer from inadequate reconstruction performance. In this
  paper, we present a novel and powerful video autoencoder capable of
  high-fidelity video encoding. First, we observe that entangling spatial and
  temporal compression by merely extending the image VAE to a 3D VAE can
  introduce motion blur and detail distortion artifacts. Thus, we propose
  temporal-aware spatial compression to better encode and decode the spatial
  information. Additionally, we integrate a lightweight motion compression model
  for further temporal compression. Second, we propose to leverage the textual
  information inherent in text-to-video datasets and incorporate text guidance
  into our model. This significantly enhances reconstruction quality,
  particularly in terms of detail preservation and temporal stability. Third, we
  further improve the versatility of our model through joint training on both
  images and videos, which not only enhances reconstruction quality but also
  enables the model to perform both image and video autoencoding. Extensive
  evaluations against strong recent baselines demonstrate the superior
  performance of our method. The project website can be found
  at~https://yzxing87.github.io/vae/{https://yzxing87.github.io/vae/}.
paper_ai_keywords: Autoencoders (VAE) temporal compression temporal-aware encoding spatial compression 3D VAE motion blur detail distortion artifacts verbal compression text-to-video datasets text guidance reconstruction quality detail preservation temporal stability joint training image autoencoding video autoencoding superior performance baselines