title: Photorealistic Video Generation with Diffusion Models
summary: We present W.A.L.T, a transformer-based approach for photorealistic video
  generation via diffusion modeling. Our approach has two key design decisions.
  First, we use a causal encoder to jointly compress images and videos within a
  unified latent space, enabling training and generation across modalities.
  Second, for memory and training efficiency, we use a window attention
  architecture tailored for joint spatial and spatiotemporal generative modeling.
  Taken together these design decisions enable us to achieve state-of-the-art
  performance on established video (UCF-101 and Kinetics-600) and image
  (ImageNet) generation benchmarks without using classifier free guidance.
  Finally, we also train a cascade of three models for the task of text-to-video
  generation consisting of a base latent video diffusion model, and two video
  super-resolution diffusion models to generate videos of 512 times 896
  resolution at 8 frames per second.
paper_authors._id: 6577f3b058d7a2cc89342628
paper_authors.name: Agrim Gupta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:17:28.178Z
paper_authors._id: 63defb9015266dd945f9f772
paper_authors.avatarUrl: /avatars/b304ac6c3b45553358f8b6cd5c3df293.svg
paper_authors.fullname: Agrim
paper_authors.type: user
paper_authors.user: agrimgupta
paper_authors._id: 6577f3b058d7a2cc89342629
paper_authors.name: Lijun Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:17:49.551Z
paper_authors._id: 639886ba65a34085ae29abb7
paper_authors.avatarUrl: /avatars/51f4d4d6fbcc07ab8b7a4c1c2f7da53d.svg
paper_authors.fullname: Lijun Yu
paper_authors.type: user
paper_authors.user: lijunyu
paper_authors._id: 6577f3b058d7a2cc8934262a
paper_authors.name: Kihyuk Sohn
paper_authors._id: 6577f3b058d7a2cc8934262b
paper_authors.name: Xiuye Gu
paper_authors._id: 6577f3b058d7a2cc8934262c
paper_authors.name: Meera Hahn
paper_authors._id: 6577f3b058d7a2cc8934262d
paper_authors.name: Li Fei-Fei
paper_authors._id: 6577f3b058d7a2cc8934262e
paper_authors.name: Irfan Essa
paper_authors._id: 6577f3b058d7a2cc8934262f
paper_authors.name: Lu Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-23T11:18:10.788Z
paper_authors._id: 64a5cba3bea0116f8f7187a7
paper_authors.avatarUrl: /avatars/97b3bad82e359c57d1ab7d6e3aa7748b.svg
paper_authors.fullname: Lu Jiang
paper_authors.type: user
paper_authors.user: roadjiang
paper_authors._id: 6577f3b058d7a2cc89342630
paper_authors.name: José Lezama
paper_title: Photorealistic Video Generation with Diffusion Models
paper_summary: We present W.A.L.T, a transformer-based approach for photorealistic video
  generation via diffusion modeling. Our approach has two key design decisions.
  First, we use a causal encoder to jointly compress images and videos within a
  unified latent space, enabling training and generation across modalities.
  Second, for memory and training efficiency, we use a window attention
  architecture tailored for joint spatial and spatiotemporal generative modeling.
  Taken together these design decisions enable us to achieve state-of-the-art
  performance on established video (UCF-101 and Kinetics-600) and image
  (ImageNet) generation benchmarks without using classifier free guidance.
  Finally, we also train a cascade of three models for the task of text-to-video
  generation consisting of a base latent video diffusion model, and two video
  super-resolution diffusion models to generate videos of 512 times 896
  resolution at 8 frames per second.
paper_ai_keywords: causal encoder unified latent space window attention architecture joint spatial and spatiotemporal generative modeling Copeland (Copeland appears to be a typo; it seems like it should be Copeland or a reference to a specific model, but since it's not clear, it's excluded) transformer-based approach diffusion modeling video generation photorealistic video generation UCF-101 Kinetics-600 ImageNet state-of-the-art performance classifier-free guidance base latent video diffusion model video super-resolution diffusion models