title: LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion
    Models
summary: This work aims to learn a high-quality text-to-video (T2V) generative model
  by leveraging a pre-trained text-to-image (T2I) model as a basis. It is a
  highly desirable yet challenging task to simultaneously a) accomplish the
  synthesis of visually realistic and temporally coherent videos while b)
  preserving the strong creative generation nature of the pre-trained T2I model.
  To this end, we propose LaVie, an integrated video generation framework that
  operates on cascaded video latent diffusion models, comprising a base T2V
  model, a temporal interpolation model, and a video super-resolution model. Our
  key insights are two-fold: 1) We reveal that the incorporation of simple
  temporal self-attentions, coupled with rotary positional encoding, adequately
  captures the temporal correlations inherent in video data. 2) Additionally, we
  validate that the process of joint image-video fine-tuning plays a pivotal role
  in producing high-quality and creative outcomes. To enhance the performance of
  LaVie, we contribute a comprehensive and diverse video dataset named Vimeo25M,
  consisting of 25 million text-video pairs that prioritize quality, diversity,
  and aesthetic appeal. Extensive experiments demonstrate that LaVie achieves
  state-of-the-art performance both quantitatively and qualitatively.
  Furthermore, we showcase the versatility of pre-trained LaVie models in various
  long video generation and personalized video synthesis applications.
paper_authors._id: 651385b6ac120f23e0053904
paper_authors.name: Yaohui Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:03:07.095Z
paper_authors._id: 63201256c6b20f03c829c4b8
paper_authors.avatarUrl: /avatars/a42092119777d65e60b12eb5ba0e45f1.svg
paper_authors.fullname: Yaohui Wang
paper_authors.type: user
paper_authors.user: YaohuiW
paper_authors._id: 651385b6ac120f23e0053905
paper_authors.name: Xinyuan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:03:16.425Z
paper_authors._id: 643e943a70c6a27621eb1c89
paper_authors.avatarUrl: /avatars/73ec521ab5ba84cc7908c52c0acef6ef.svg
paper_authors.fullname: Xinyuan Chen
paper_authors.type: user
paper_authors.user: AriaChen
paper_authors._id: 651385b6ac120f23e0053906
paper_authors.name: Xin Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-28T08:20:25.748Z
paper_authors._id: 64a51faf9ec0167a99a4d114
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a51faf9ec0167a99a4d114/IlJS0mwkF_ZIebaQrbJIl.jpeg
paper_authors.fullname: Xin Ma
paper_authors.type: user
paper_authors.user: maxin-cn
paper_authors._id: 651385b6ac120f23e0053907
paper_authors.name: Shangchen Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:04:30.137Z
paper_authors._id: 62e57662ae9d3f10acbb1b1b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e57662ae9d3f10acbb1b1b/lg58jdbNyv6LGH2LFnZDF.png
paper_authors.fullname: Shangchen Zhou
paper_authors.type: user
paper_authors.user: sczhou
paper_authors._id: 651385b6ac120f23e0053908
paper_authors.name: Ziqi Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-27T07:26:53.301Z
paper_authors._id: 60efe7fa0d920bc7805cada5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png
paper_authors.fullname: Ziqi Huang
paper_authors.type: user
paper_authors.user: Ziqi
paper_authors._id: 651385b6ac120f23e0053909
paper_authors.name: Yi Wang
paper_authors._id: 651385b6ac120f23e005390a
paper_authors.name: Ceyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:05:59.235Z
paper_authors._id: 64accf3fc1da7c4dbc69392d
paper_authors.avatarUrl: /avatars/ec8ad2498f8a49a7a00c77a7f70d34bb.svg
paper_authors.fullname: Ceyuan
paper_authors.type: user
paper_authors.user: Ceyuan
paper_authors._id: 651385b6ac120f23e005390b
paper_authors.name: Yinan He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-09T07:53:14.524Z
paper_authors._id: 62aafa49f29ff279b51f0182
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
paper_authors.fullname: yinanhe
paper_authors.type: user
paper_authors.user: ynhe
paper_authors._id: 651385b6ac120f23e005390c
paper_authors.name: Jiashuo Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:06:21.215Z
paper_authors._id: 64b89a14cf14c2fabe96664c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
paper_authors.fullname: Jiashuo Yu
paper_authors.type: user
paper_authors.user: awojustin
paper_authors._id: 651385b6ac120f23e005390d
paper_authors.name: Peiqing Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:06:28.910Z
paper_authors._id: 6513aae6330c55fdc5462ca8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EDhpiTqCBMNPmMGrOKcvY.jpeg
paper_authors.fullname: pq-yang
paper_authors.type: user
paper_authors.user: PeiqingYang
paper_authors._id: 651385b6ac120f23e005390e
paper_authors.name: Yuwei Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:08:47.559Z
paper_authors._id: 6371f83d5ffa922d638ef486
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6371f83d5ffa922d638ef486/UzWl3xhVvtol3Rz1mjQ9t.jpeg
paper_authors.fullname: Yuwei Guo
paper_authors.type: user
paper_authors.user: guoyww
paper_authors._id: 651385b6ac120f23e005390f
paper_authors.name: Tianxing Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:08:56.931Z
paper_authors._id: 63185a8973e9e4aa51dbc0ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676359176854-63185a8973e9e4aa51dbc0ce.jpeg
paper_authors.fullname: Tianxing Wu
paper_authors.type: user
paper_authors.user: TianxingWu
paper_authors._id: 651385b6ac120f23e0053910
paper_authors.name: Chenyang Si
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:09:04.159Z
paper_authors._id: 635f8ed47c05eb9f59963d3a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
paper_authors.fullname: ChenyangSi
paper_authors.type: user
paper_authors.user: ChenyangSi
paper_authors._id: 651385b6ac120f23e0053911
paper_authors.name: Yuming Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:09:11.257Z
paper_authors._id: 629c95b7a5d6f5fe10e6ed45
paper_authors.avatarUrl: /avatars/c9a5f0e7fae3a079d26db266e9ff90a3.svg
paper_authors.fullname: Yuming Jiang
paper_authors.type: user
paper_authors.user: yumingj
paper_authors._id: 651385b6ac120f23e0053912
paper_authors.name: Cunjian Chen
paper_authors._id: 651385b6ac120f23e0053913
paper_authors.name: Chen Change Loy
paper_authors._id: 651385b6ac120f23e0053914
paper_authors.name: Bo Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:09:34.383Z
paper_authors._id: 635f93577c05eb9f59966209
paper_authors.avatarUrl: /avatars/add48d7e3790a6b500d6c451ef8b0f75.svg
paper_authors.fullname: Intelligent Digital Creation
paper_authors.type: user
paper_authors.user: BoDai
paper_authors._id: 651385b6ac120f23e0053915
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:09:42.739Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 651385b6ac120f23e0053916
paper_authors.name: Yu Qiao
paper_authors._id: 651385b6ac120f23e0053917
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:10:55.550Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion
    Models
paper_summary: This work aims to learn a high-quality text-to-video (T2V) generative model
  by leveraging a pre-trained text-to-image (T2I) model as a basis. It is a
  highly desirable yet challenging task to simultaneously a) accomplish the
  synthesis of visually realistic and temporally coherent videos while b)
  preserving the strong creative generation nature of the pre-trained T2I model.
  To this end, we propose LaVie, an integrated video generation framework that
  operates on cascaded video latent diffusion models, comprising a base T2V
  model, a temporal interpolation model, and a video super-resolution model. Our
  key insights are two-fold: 1) We reveal that the incorporation of simple
  temporal self-attentions, coupled with rotary positional encoding, adequately
  captures the temporal correlations inherent in video data. 2) Additionally, we
  validate that the process of joint image-video fine-tuning plays a pivotal role
  in producing high-quality and creative outcomes. To enhance the performance of
  LaVie, we contribute a comprehensive and diverse video dataset named Vimeo25M,
  consisting of 25 million text-video pairs that prioritize quality, diversity,
  and aesthetic appeal. Extensive experiments demonstrate that LaVie achieves
  state-of-the-art performance both quantitatively and qualitatively.
  Furthermore, we showcase the versatility of pre-trained LaVie models in various
  long video generation and personalized video synthesis applications.
paper_ai_summary: LaVie, a text-to-video framework integrating cascaded video latent diffusion models, achieves state-of-the-art video generation by leveraging temporal self-attentions and joint image-video fine-tuning.
paper_ai_keywords: text-to-video text-to-image latent diffusion models temporal self-attentions rotary positional encoding video super-resolution joint image-video fine-tuning Vimeo25M