WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent
  Video Diffusion Model
Video Variational Autoencoder (VAE) encodes videos into a low-dimensional
latent space, becoming a key component of most Latent Video Diffusion Models
(LVDMs) to reduce model training costs. However, as the resolution and duration
of generated videos increase, the encoding cost of Video VAEs becomes a
limiting bottleneck in training LVDMs. Moreover, the block-wise inference
method adopted by most LVDMs can lead to discontinuities of latent space when
processing long-duration videos. The key to addressing the computational
bottleneck lies in decomposing videos into distinct components and efficiently
encoding the critical information. Wavelet transform can decompose videos into
multiple frequency-domain components and improve the efficiency significantly,
we thus propose Wavelet Flow VAE (WF-VAE), an autoencoder that leverages
multi-level wavelet transform to facilitate low-frequency energy flow into
latent representation. Furthermore, we introduce a method called Causal Cache,
which maintains the integrity of latent space during block-wise inference.
Compared to state-of-the-art video VAEs, WF-VAE demonstrates superior
performance in both PSNR and LPIPS metrics, achieving 2x higher throughput and
4x lower memory consumption while maintaining competitive reconstruction
quality. Our code and models are available at
https://github.com/PKU-YuanGroup/WF-VAE.
674941ae421f757855b3d3d9
Zongjian Li
admin_assigned
2024-12-03T15:23:21.964Z
64495f0616425b79ffeebb73
https://cdn-avatars.huggingface.co/v1/production/uploads/64495f0616425b79ffeebb73/KAwrRqiAiXzV7arOFVQQZ.jpeg
Zong-jian Li
user
ZongjianLi
674941ae421f757855b3d3da
Bin Lin
claimed_verified
2024-12-03T11:16:29.796Z
6367a8175bb06007ea099b8f
https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
linbin
user
LanguageBind
674941ae421f757855b3d3db
Yang Ye
674941ae421f757855b3d3dc
Liuhan Chen
admin_assigned
2024-12-03T15:23:30.042Z
65c0588569429d85dc594a85
/avatars/822203d982519b999247a53235e30c39.svg
liuhan
user
LiuhanChen
674941ae421f757855b3d3dd
Xinhua Cheng
admin_assigned
2024-12-03T15:23:38.006Z
645b4026f9d4ec91fdd516ce
/avatars/fb6556ab973764af5c44980f4e8fcc71.svg
Xinhua Cheng
user
cxh0519
674941ae421f757855b3d3de
Shenghai Yuan
claimed_verified
2024-12-03T11:16:31.845Z
63468720dd6d90d82ccf3450
https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
YSH
user
BestWishYsh
674941ae421f757855b3d3df
Li Yuan
admin_assigned
2024-12-03T15:23:44.798Z
614030bd8cbdb613b82f36a8
https://cdn-avatars.huggingface.co/v1/production/uploads/1631596713749-noauth.jpeg
Li Yuan
user
LiYuan
WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent
  Video Diffusion Model
Video Variational Autoencoder (VAE) encodes videos into a low-dimensional
latent space, becoming a key component of most Latent Video Diffusion Models
(LVDMs) to reduce model training costs. However, as the resolution and duration
of generated videos increase, the encoding cost of Video VAEs becomes a
limiting bottleneck in training LVDMs. Moreover, the block-wise inference
method adopted by most LVDMs can lead to discontinuities of latent space when
processing long-duration videos. The key to addressing the computational
bottleneck lies in decomposing videos into distinct components and efficiently
encoding the critical information. Wavelet transform can decompose videos into
multiple frequency-domain components and improve the efficiency significantly,
we thus propose Wavelet Flow VAE (WF-VAE), an autoencoder that leverages
multi-level wavelet transform to facilitate low-frequency energy flow into
latent representation. Furthermore, we introduce a method called Causal Cache,
which maintains the integrity of latent space during block-wise inference.
Compared to state-of-the-art video VAEs, WF-VAE demonstrates superior
performance in both PSNR and LPIPS metrics, achieving 2x higher throughput and
4x lower memory consumption while maintaining competitive reconstruction
quality. Our code and models are available at
https://github.com/PKU-YuanGroup/WF-VAE.
Wavelet Flow VAE uses multi-level wavelet transform and Causal Cache to enhance video VAE performance, reducing computational costs and memory usage while maintaining reconstruction quality.
Video VAE Latent Video Diffusion Models LVDMs Wavelet transform WF-VAE multi-level wavelet transform low-frequency energy flow Causal Cache PSNR LPIPS