title: CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
summary: We introduce CogVideoX, a large-scale diffusion transformer model designed
  for generating videos based on text prompts. To efficently model video data, we
  propose to levearge a 3D Variational Autoencoder (VAE) to compress videos along
  both spatial and temporal dimensions. To improve the text-video alignment, we
  propose an expert transformer with the expert adaptive LayerNorm to facilitate
  the deep fusion between the two modalities. By employing a progressive training
  technique, CogVideoX is adept at producing coherent, long-duration videos
  characterized by significant motions. In addition, we develop an effective
  text-video data processing pipeline that includes various data preprocessing
  strategies and a video captioning method. It significantly helps enhance the
  performance of CogVideoX, improving both generation quality and semantic
  alignment. Results show that CogVideoX demonstrates state-of-the-art
  performance across both multiple machine metrics and human evaluations. The
  model weights of both the 3D Causal VAE and CogVideoX are publicly available at
  https://github.com/THUDM/CogVideo.
paper_authors._id: 66bacf59c9b2ab14b398f761
paper_authors.name: Zhuoyi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T09:11:01.293Z
paper_authors._id: 6322e02496d6c8518eb05292
paper_authors.avatarUrl: /avatars/6b5d317f9ccb0046e40918c7861b4cb5.svg
paper_authors.fullname: zhuoyi
paper_authors.type: user
paper_authors.user: keg-yzy
paper_authors._id: 66bacf59c9b2ab14b398f762
paper_authors.name: Jiayan Teng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:45:51.899Z
paper_authors._id: 65228733377bffdc59a10117
paper_authors.avatarUrl: /avatars/6eec07553658ab22f8058caa0bfbed49.svg
paper_authors.fullname: tengjiayan
paper_authors.type: user
paper_authors.user: tengjiayan
paper_authors._id: 66bacf59c9b2ab14b398f763
paper_authors.name: Wendi Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:47:07.561Z
paper_authors._id: 650aaaf1634e02df56dfd231
paper_authors.avatarUrl: /avatars/4643ab23721d4ed6aeb1ebbc717adc43.svg
paper_authors.fullname: Wendi Zheng
paper_authors.type: user
paper_authors.user: zwd125
paper_authors._id: 66bacf59c9b2ab14b398f764
paper_authors.name: Ming Ding
paper_authors._id: 66bacf59c9b2ab14b398f765
paper_authors.name: Shiyu Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:47:33.688Z
paper_authors._id: 6406db5cd684369027166986
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6406db5cd684369027166986/Zl-orrGcbY0RbfjfKszn1.jpeg
paper_authors.fullname: Shiyu Huang
paper_authors.type: user
paper_authors.user: ShiyuHuang
paper_authors._id: 66bacf59c9b2ab14b398f766
paper_authors.name: Jiazheng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:47:56.691Z
paper_authors._id: 62d7b131f6e8ba66107af761
paper_authors.avatarUrl: /avatars/f1c5df47aef69c824fd166722df8f670.svg
paper_authors.fullname: Jiazheng Xu
paper_authors.type: user
paper_authors.user: xujz0703
paper_authors._id: 66bacf59c9b2ab14b398f767
paper_authors.name: Yuanming Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:48:10.864Z
paper_authors._id: 659b6c50b0f43ed69fe09d56
paper_authors.avatarUrl: /avatars/8ea56c56263595a9f7555f2c2520641a.svg
paper_authors.fullname: 杨远明
paper_authors.type: user
paper_authors.user: yangyuanming
paper_authors._id: 66bacf59c9b2ab14b398f768
paper_authors.name: Wenyi Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:48:17.739Z
paper_authors._id: 62ecd24cb8764c7738ef2793
paper_authors.avatarUrl: /avatars/c1b80b5c55f9d652c1aaac7919e1fa32.svg
paper_authors.fullname: Wenyi Hong
paper_authors.type: user
paper_authors.user: wenyi
paper_authors._id: 66bacf59c9b2ab14b398f769
paper_authors.name: Xiaohan Zhang
paper_authors._id: 66bacf59c9b2ab14b398f76a
paper_authors.name: Guanyu Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:48:57.709Z
paper_authors._id: 64d996d6bcab729cb400cb70
paper_authors.avatarUrl: /avatars/3db57c8b643fba8302ed39d8cf0f4ddb.svg
paper_authors.fullname: Guanyu Feng
paper_authors.type: user
paper_authors.user: jiguanglizipao
paper_authors._id: 66bacf59c9b2ab14b398f76b
paper_authors.name: Da Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:49:04.537Z
paper_authors._id: 634e4670a51d5df8c2d92fce
paper_authors.avatarUrl: /avatars/c52d7150b4de6a2eb2d83b345d35cbc2.svg
paper_authors.fullname: Da Yin
paper_authors.type: user
paper_authors.user: DaYin
paper_authors._id: 66bacf59c9b2ab14b398f76c
paper_authors.name: Xiaotao Gu
paper_authors._id: 66bacf59c9b2ab14b398f76d
paper_authors.name: Yuxuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T15:29:07.523Z
paper_authors._id: 643507d1ce04fdb57e9d7e05
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643507d1ce04fdb57e9d7e05/QuCFAO3v7G3LrVGusqFj1.png
paper_authors.fullname: zR
paper_authors.type: user
paper_authors.user: zRzRzRzRzRzRzR
paper_authors._id: 66bacf59c9b2ab14b398f76e
paper_authors.name: Weihan Wang
paper_authors._id: 66bacf59c9b2ab14b398f76f
paper_authors.name: Yean Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:50:14.803Z
paper_authors._id: 65acc5afe2a2c8635614de43
paper_authors.avatarUrl: /avatars/c5fce792792cc0b52ed7475d72460c58.svg
paper_authors.fullname: Yean Cheng
paper_authors.type: user
paper_authors.user: LiquidAmmonia
paper_authors._id: 66bacf59c9b2ab14b398f770
paper_authors.name: Ting Liu
paper_authors._id: 66bacf59c9b2ab14b398f771
paper_authors.name: Bin Xu
paper_authors._id: 66bacf59c9b2ab14b398f772
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:51:14.639Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_authors._id: 66bacf59c9b2ab14b398f773
paper_authors.name: Jie Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:51:32.360Z
paper_authors._id: 640dff05474aa6f89556677e
paper_authors.avatarUrl: /avatars/1b4591c7322d649c797b3125148f1915.svg
paper_authors.fullname: Jie Tang
paper_authors.type: user
paper_authors.user: jerytang
paper_title: CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
paper_summary: We introduce CogVideoX, a large-scale diffusion transformer model designed
  for generating videos based on text prompts. To efficently model video data, we
  propose to levearge a 3D Variational Autoencoder (VAE) to compress videos along
  both spatial and temporal dimensions. To improve the text-video alignment, we
  propose an expert transformer with the expert adaptive LayerNorm to facilitate
  the deep fusion between the two modalities. By employing a progressive training
  technique, CogVideoX is adept at producing coherent, long-duration videos
  characterized by significant motions. In addition, we develop an effective
  text-video data processing pipeline that includes various data preprocessing
  strategies and a video captioning method. It significantly helps enhance the
  performance of CogVideoX, improving both generation quality and semantic
  alignment. Results show that CogVideoX demonstrates state-of-the-art
  performance across both multiple machine metrics and human evaluations. The
  model weights of both the 3D Causal VAE and CogVideoX are publicly available at
  https://github.com/THUDM/CogVideo.
paper_ai_keywords: diffusion transformer 3D Variational Autoencoder (VAE) expert transformer expert adaptive LayerNorm text-video alignment image captioning machine metrics human evaluations Causal VAE