VidTwin: Video VAE with Decoupled Structure and Dynamics
Recent advancements in video autoencoders (Video AEs) have significantly
improved the quality and efficiency of video generation. In this paper, we
propose a novel and compact video autoencoder, VidTwin, that decouples video
into two distinct latent spaces: Structure latent vectors, which capture
overall content and global movement, and Dynamics latent vectors, which
represent fine-grained details and rapid movements. Specifically, our approach
leverages an Encoder-Decoder backbone, augmented with two submodules for
extracting these latent spaces, respectively. The first submodule employs a
Q-Former to extract low-frequency motion trends, followed by downsampling
blocks to remove redundant content details. The second averages the latent
vectors along the spatial dimension to capture rapid motion. Extensive
experiments show that VidTwin achieves a high compression rate of 0.20% with
high reconstruction quality (PSNR of 28.14 on the MCL-JCV dataset), and
performs efficiently and effectively in downstream generative tasks. Moreover,
our model demonstrates explainability and scalability, paving the way for
future research in video latent representation and generation. Our code has
been released at https://github.com/microsoft/VidTok/tree/main/vidtwin.
676d52f991c1773322dcc757
Yuchi Wang
claimed_verified
2024-12-30T19:31:48.752Z
622842e296588dd1a2594746
/avatars/b96d0b49e4cff25d83fefc67b7cd1076.svg
wangyuchi
user
YuchiWang
676d52f991c1773322dcc758
Junliang Guo
admin_assigned
2024-12-30T21:17:48.498Z
66868659ccb9539da85c4e14
/avatars/515a49363872c23d57a6f75063606348.svg
Junliang Guo
user
leo-guo
676d52f991c1773322dcc759
Xinyi Xie
admin_assigned
2024-12-30T21:17:54.976Z
643cf31cb561f17fa4678293
/avatars/dd02c667d0d00704fb41c7b7065dce0c.svg
Xinyi XIE
user
Hyude
676d52f991c1773322dcc75a
Tianyu He
676d52f991c1773322dcc75b
Xu Sun
676d52f991c1773322dcc75c
Jiang Bian
admin_assigned
2024-12-30T21:18:06.605Z
63f253f8f4e30ffd2bd308fb
/avatars/303f4c7ee588f638acf78a7966786e1e.svg
Jiang Bian
user
bianjiang
VidTwin: Video VAE with Decoupled Structure and Dynamics
Recent advancements in video autoencoders (Video AEs) have significantly
improved the quality and efficiency of video generation. In this paper, we
propose a novel and compact video autoencoder, VidTwin, that decouples video
into two distinct latent spaces: Structure latent vectors, which capture
overall content and global movement, and Dynamics latent vectors, which
represent fine-grained details and rapid movements. Specifically, our approach
leverages an Encoder-Decoder backbone, augmented with two submodules for
extracting these latent spaces, respectively. The first submodule employs a
Q-Former to extract low-frequency motion trends, followed by downsampling
blocks to remove redundant content details. The second averages the latent
vectors along the spatial dimension to capture rapid motion. Extensive
experiments show that VidTwin achieves a high compression rate of 0.20% with
high reconstruction quality (PSNR of 28.14 on the MCL-JCV dataset), and
performs efficiently and effectively in downstream generative tasks. Moreover,
our model demonstrates explainability and scalability, paving the way for
future research in video latent representation and generation. Our code has
been released at https://github.com/microsoft/VidTok/tree/main/vidtwin.
VidTwin, a novel video autoencoder, decouples video into structure and dynamics latent spaces using a Q-Former and downsampling, achieving high compression and reconstruction quality.
video autoencoders VidTwin latent spaces structure latent vectors dynamics latent vectors Encoder-Decoder Q-Former downsampling PSNR MCL-JCV dataset generative tasks explainability scalability