EasyAnimate: A High-Performance Long Video Generation Method based on
  Transformer Architecture
This paper presents EasyAnimate, an advanced method for video generation that
leverages the power of transformer architecture for high-performance outcomes.
We have expanded the DiT framework originally designed for 2D image synthesis
to accommodate the complexities of 3D video generation by incorporating a
motion module block. It is used to capture temporal dynamics, thereby ensuring
the production of consistent frames and seamless motion transitions. The motion
module can be adapted to various DiT baseline methods to generate video with
different styles. It can also generate videos with different frame rates and
resolutions during both training and inference phases, suitable for both images
and videos. Moreover, we introduce slice VAE, a novel approach to condense the
temporal axis, facilitating the generation of long duration videos. Currently,
EasyAnimate exhibits the proficiency to generate videos with 144 frames. We
provide a holistic ecosystem for video production based on DiT, encompassing
aspects such as data pre-processing, VAE training, DiT models training (both
the baseline model and LoRA model), and end-to-end video inference. Code is
available at: https://github.com/aigc-apps/EasyAnimate. We are continuously
working to enhance the performance of our method.
66588c57893215131946bea6
Jiaqi Xu
66588c57893215131946bea7
Xinyi Zou
admin_assigned
2024-05-31T10:50:39.399Z
6419b3e98e0baaeed5a18333
/avatars/915bc1d711630c78c0c2760ebdf104b7.svg
Xinyi Zou
user
Zouxinyi
66588c57893215131946bea8
Kunzhe Huang
admin_assigned
2024-05-31T10:50:45.742Z
64c22cd08d3481788e212e9c
/avatars/4484668c1090990ae678050bc07f9a68.svg
Kunzhe Huang
user
hkunzhe
66588c57893215131946bea9
Yunkuo Chen
admin_assigned
2024-05-31T10:50:27.607Z
62009887577fcfa0ce82f1b0
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/laLRCNcbJc2bZzf_4MCXR.png
chenyunkuo
user
yunkchen
66588c57893215131946beaa
Bo Liu
66588c57893215131946beab
MengLi Cheng
66588c57893215131946beac
Xing Shi
66588c57893215131946bead
Jun Huang
EasyAnimate: A High-Performance Long Video Generation Method based on
  Transformer Architecture
This paper presents EasyAnimate, an advanced method for video generation that
leverages the power of transformer architecture for high-performance outcomes.
We have expanded the DiT framework originally designed for 2D image synthesis
to accommodate the complexities of 3D video generation by incorporating a
motion module block. It is used to capture temporal dynamics, thereby ensuring
the production of consistent frames and seamless motion transitions. The motion
module can be adapted to various DiT baseline methods to generate video with
different styles. It can also generate videos with different frame rates and
resolutions during both training and inference phases, suitable for both images
and videos. Moreover, we introduce slice VAE, a novel approach to condense the
temporal axis, facilitating the generation of long duration videos. Currently,
EasyAnimate exhibits the proficiency to generate videos with 144 frames. We
provide a holistic ecosystem for video production based on DiT, encompassing
aspects such as data pre-processing, VAE training, DiT models training (both
the baseline model and LoRA model), and end-to-end video inference. Code is
available at: https://github.com/aigc-apps/EasyAnimate. We are continuously
working to enhance the performance of our method.
EasyAnimate leverages transformer architecture to generate 3D videos with consistent motion and varying frame rates and resolutions using a motion module and slice VAE.
transformer architecture DiT framework motion module block temporal dynamics slice VAE video inference LoRA model