title: WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent
    Video Diffusion Model
summary: Video Variational Autoencoder (VAE) encodes videos into a low-dimensional
  latent space, becoming a key component of most Latent Video Diffusion Models
  (LVDMs) to reduce model training costs. However, as the resolution and duration
  of generated videos increase, the encoding cost of Video VAEs becomes a
  limiting bottleneck in training LVDMs. Moreover, the block-wise inference
  method adopted by most LVDMs can lead to discontinuities of latent space when
  processing long-duration videos. The key to addressing the computational
  bottleneck lies in decomposing videos into distinct components and efficiently
  encoding the critical information. Wavelet transform can decompose videos into
  multiple frequency-domain components and improve the efficiency significantly,
  we thus propose Wavelet Flow VAE (WF-VAE), an autoencoder that leverages
  multi-level wavelet transform to facilitate low-frequency energy flow into
  latent representation. Furthermore, we introduce a method called Causal Cache,
  which maintains the integrity of latent space during block-wise inference.
  Compared to state-of-the-art video VAEs, WF-VAE demonstrates superior
  performance in both PSNR and LPIPS metrics, achieving 2x higher throughput and
  4x lower memory consumption while maintaining competitive reconstruction
  quality. Our code and models are available at
  https://github.com/PKU-YuanGroup/WF-VAE.
paper_authors._id: 674941ae421f757855b3d3d9
paper_authors.name: Zongjian Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:23:21.964Z
paper_authors._id: 64495f0616425b79ffeebb73
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64495f0616425b79ffeebb73/KAwrRqiAiXzV7arOFVQQZ.jpeg
paper_authors.fullname: Zong-jian Li
paper_authors.type: user
paper_authors.user: ZongjianLi
paper_authors._id: 674941ae421f757855b3d3da
paper_authors.name: Bin Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:16:29.796Z
paper_authors._id: 6367a8175bb06007ea099b8f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
paper_authors.fullname: linbin
paper_authors.type: user
paper_authors.user: LanguageBind
paper_authors._id: 674941ae421f757855b3d3db
paper_authors.name: Yang Ye
paper_authors._id: 674941ae421f757855b3d3dc
paper_authors.name: Liuhan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:23:30.042Z
paper_authors._id: 65c0588569429d85dc594a85
paper_authors.avatarUrl: /avatars/822203d982519b999247a53235e30c39.svg
paper_authors.fullname: liuhan
paper_authors.type: user
paper_authors.user: LiuhanChen
paper_authors._id: 674941ae421f757855b3d3dd
paper_authors.name: Xinhua Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:23:38.006Z
paper_authors._id: 645b4026f9d4ec91fdd516ce
paper_authors.avatarUrl: /avatars/fb6556ab973764af5c44980f4e8fcc71.svg
paper_authors.fullname: Xinhua Cheng
paper_authors.type: user
paper_authors.user: cxh0519
paper_authors._id: 674941ae421f757855b3d3de
paper_authors.name: Shenghai Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:16:31.845Z
paper_authors._id: 63468720dd6d90d82ccf3450
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
paper_authors.fullname: YSH
paper_authors.type: user
paper_authors.user: BestWishYsh
paper_authors._id: 674941ae421f757855b3d3df
paper_authors.name: Li Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:23:44.798Z
paper_authors._id: 614030bd8cbdb613b82f36a8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631596713749-noauth.jpeg
paper_authors.fullname: Li Yuan
paper_authors.type: user
paper_authors.user: LiYuan
paper_title: WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent
    Video Diffusion Model
paper_summary: Video Variational Autoencoder (VAE) encodes videos into a low-dimensional
  latent space, becoming a key component of most Latent Video Diffusion Models
  (LVDMs) to reduce model training costs. However, as the resolution and duration
  of generated videos increase, the encoding cost of Video VAEs becomes a
  limiting bottleneck in training LVDMs. Moreover, the block-wise inference
  method adopted by most LVDMs can lead to discontinuities of latent space when
  processing long-duration videos. The key to addressing the computational
  bottleneck lies in decomposing videos into distinct components and efficiently
  encoding the critical information. Wavelet transform can decompose videos into
  multiple frequency-domain components and improve the efficiency significantly,
  we thus propose Wavelet Flow VAE (WF-VAE), an autoencoder that leverages
  multi-level wavelet transform to facilitate low-frequency energy flow into
  latent representation. Furthermore, we introduce a method called Causal Cache,
  which maintains the integrity of latent space during block-wise inference.
  Compared to state-of-the-art video VAEs, WF-VAE demonstrates superior
  performance in both PSNR and LPIPS metrics, achieving 2x higher throughput and
  4x lower memory consumption while maintaining competitive reconstruction
  quality. Our code and models are available at
  https://github.com/PKU-YuanGroup/WF-VAE.
paper_ai_keywords: Video VAE Latent Video Diffusion Models (LVDMs) Latent space block-wise inference Wavelet transform multi-level wavelet transform Wavelet Flow VAE (WF-VAE) Causal Cache PSNR LPIPS