title: VidTwin: Video VAE with Decoupled Structure and Dynamics
summary: Recent advancements in video autoencoders (Video AEs) have significantly
  improved the quality and efficiency of video generation. In this paper, we
  propose a novel and compact video autoencoder, VidTwin, that decouples video
  into two distinct latent spaces: Structure latent vectors, which capture
  overall content and global movement, and Dynamics latent vectors, which
  represent fine-grained details and rapid movements. Specifically, our approach
  leverages an Encoder-Decoder backbone, augmented with two submodules for
  extracting these latent spaces, respectively. The first submodule employs a
  Q-Former to extract low-frequency motion trends, followed by downsampling
  blocks to remove redundant content details. The second averages the latent
  vectors along the spatial dimension to capture rapid motion. Extensive
  experiments show that VidTwin achieves a high compression rate of 0.20% with
  high reconstruction quality (PSNR of 28.14 on the MCL-JCV dataset), and
  performs efficiently and effectively in downstream generative tasks. Moreover,
  our model demonstrates explainability and scalability, paving the way for
  future research in video latent representation and generation. Our code has
  been released at https://github.com/microsoft/VidTok/tree/main/vidtwin.
paper_authors._id: 676d52f991c1773322dcc757
paper_authors.name: Yuchi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:31:48.752Z
paper_authors._id: 622842e296588dd1a2594746
paper_authors.avatarUrl: /avatars/b96d0b49e4cff25d83fefc67b7cd1076.svg
paper_authors.fullname: wangyuchi
paper_authors.type: user
paper_authors.user: YuchiWang
paper_authors._id: 676d52f991c1773322dcc758
paper_authors.name: Junliang Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:17:48.498Z
paper_authors._id: 66868659ccb9539da85c4e14
paper_authors.avatarUrl: /avatars/515a49363872c23d57a6f75063606348.svg
paper_authors.fullname: Junliang Guo
paper_authors.type: user
paper_authors.user: leo-guo
paper_authors._id: 676d52f991c1773322dcc759
paper_authors.name: Xinyi Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:17:54.976Z
paper_authors._id: 643cf31cb561f17fa4678293
paper_authors.avatarUrl: /avatars/dd02c667d0d00704fb41c7b7065dce0c.svg
paper_authors.fullname: Xinyi XIE
paper_authors.type: user
paper_authors.user: Hyude
paper_authors._id: 676d52f991c1773322dcc75a
paper_authors.name: Tianyu He
paper_authors._id: 676d52f991c1773322dcc75b
paper_authors.name: Xu Sun
paper_authors._id: 676d52f991c1773322dcc75c
paper_authors.name: Jiang Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:18:06.605Z
paper_authors._id: 63f253f8f4e30ffd2bd308fb
paper_authors.avatarUrl: /avatars/303f4c7ee588f638acf78a7966786e1e.svg
paper_authors.fullname: Jiang Bian
paper_authors.type: user
paper_authors.user: bianjiang
paper_title: VidTwin: Video VAE with Decoupled Structure and Dynamics
paper_summary: Recent advancements in video autoencoders (Video AEs) have significantly
  improved the quality and efficiency of video generation. In this paper, we
  propose a novel and compact video autoencoder, VidTwin, that decouples video
  into two distinct latent spaces: Structure latent vectors, which capture
  overall content and global movement, and Dynamics latent vectors, which
  represent fine-grained details and rapid movements. Specifically, our approach
  leverages an Encoder-Decoder backbone, augmented with two submodules for
  extracting these latent spaces, respectively. The first submodule employs a
  Q-Former to extract low-frequency motion trends, followed by downsampling
  blocks to remove redundant content details. The second averages the latent
  vectors along the spatial dimension to capture rapid motion. Extensive
  experiments show that VidTwin achieves a high compression rate of 0.20% with
  high reconstruction quality (PSNR of 28.14 on the MCL-JCV dataset), and
  performs efficiently and effectively in downstream generative tasks. Moreover,
  our model demonstrates explainability and scalability, paving the way for
  future research in video latent representation and generation. Our code has
  been released at https://github.com/microsoft/VidTok/tree/main/vidtwin.
paper_ai_keywords: video autoencoders (Video AEs) VidTwin Structure latent vectors Dynamics latent vectors Encoder-Decoder backbone Q-Former downsampling blocks latent vectors PSNR MCL-JCV dataset compression rate reconstruction quality explainability scalability video latent representation video generation