Large Motion Video Autoencoding with Cross-modal Video VAE
Learning a robust video Variational Autoencoder (VAE) is essential for
reducing video redundancy and facilitating efficient video generation. Directly
applying image VAEs to individual frames in isolation can result in temporal
inconsistencies and suboptimal compression rates due to a lack of temporal
compression. Existing Video VAEs have begun to address temporal compression;
however, they often suffer from inadequate reconstruction performance. In this
paper, we present a novel and powerful video autoencoder capable of
high-fidelity video encoding. First, we observe that entangling spatial and
temporal compression by merely extending the image VAE to a 3D VAE can
introduce motion blur and detail distortion artifacts. Thus, we propose
temporal-aware spatial compression to better encode and decode the spatial
information. Additionally, we integrate a lightweight motion compression model
for further temporal compression. Second, we propose to leverage the textual
information inherent in text-to-video datasets and incorporate text guidance
into our model. This significantly enhances reconstruction quality,
particularly in terms of detail preservation and temporal stability. Third, we
further improve the versatility of our model through joint training on both
images and videos, which not only enhances reconstruction quality but also
enables the model to perform both image and video autoencoding. Extensive
evaluations against strong recent baselines demonstrate the superior
performance of our method. The project website can be found
at~https://yzxing87.github.io/vae/{https://yzxing87.github.io/vae/}.
676a5396207235e4b020dfda
Yazhou Xing
676a5396207235e4b020dfdb
Yang Fei
claimed_verified
2025-02-04T09:39:56.502Z
648ffe95669191ccb6772a2e
/avatars/025cac7dc64ea9ef2074754c92086baa.svg
Yang Fei
user
sunfly
676a5396207235e4b020dfdc
Yingqing He
676a5396207235e4b020dfdd
Jingye Chen
claimed_verified
2025-01-03T14:05:38.574Z
6478a982256b62e219917d67
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/PUJ-N2cQxgEmDGfyjajyA.jpeg
JingyeChen22
user
JingyeChen22
676a5396207235e4b020dfde
Jiaxin Xie
676a5396207235e4b020dfdf
Xiaowei Chi
676a5396207235e4b020dfe0
Qifeng Chen
Large Motion Video Autoencoding with Cross-modal Video VAE
Learning a robust video Variational Autoencoder (VAE) is essential for
reducing video redundancy and facilitating efficient video generation. Directly
applying image VAEs to individual frames in isolation can result in temporal
inconsistencies and suboptimal compression rates due to a lack of temporal
compression. Existing Video VAEs have begun to address temporal compression;
however, they often suffer from inadequate reconstruction performance. In this
paper, we present a novel and powerful video autoencoder capable of
high-fidelity video encoding. First, we observe that entangling spatial and
temporal compression by merely extending the image VAE to a 3D VAE can
introduce motion blur and detail distortion artifacts. Thus, we propose
temporal-aware spatial compression to better encode and decode the spatial
information. Additionally, we integrate a lightweight motion compression model
for further temporal compression. Second, we propose to leverage the textual
information inherent in text-to-video datasets and incorporate text guidance
into our model. This significantly enhances reconstruction quality,
particularly in terms of detail preservation and temporal stability. Third, we
further improve the versatility of our model through joint training on both
images and videos, which not only enhances reconstruction quality but also
enables the model to perform both image and video autoencoding. Extensive
evaluations against strong recent baselines demonstrate the superior
performance of our method. The project website can be found
at~https://yzxing87.github.io/vae/{https://yzxing87.github.io/vae/}.
A novel video Variational Autoencoder (VAE) integrates temporal-aware spatial compression and motion compression with text guidance to achieve high-fidelity video encoding and superior reconstruction quality.
Video VAE temporal compression motion blur detail distortion artifacts temporal-aware spatial compression lightweight motion compression model text-to-video datasets text guidance high-fidelity video encoding joint training image autoencoding video autoencoding