title: Lumiere: A Space-Time Diffusion Model for Video Generation
summary: We introduce Lumiere -- a text-to-video diffusion model designed for
  synthesizing videos that portray realistic, diverse and coherent motion -- a
  pivotal challenge in video synthesis. To this end, we introduce a Space-Time
  U-Net architecture that generates the entire temporal duration of the video at
  once, through a single pass in the model. This is in contrast to existing video
  models which synthesize distant keyframes followed by temporal super-resolution
  -- an approach that inherently makes global temporal consistency difficult to
  achieve. By deploying both spatial and (importantly) temporal down- and
  up-sampling and leveraging a pre-trained text-to-image diffusion model, our
  model learns to directly generate a full-frame-rate, low-resolution video by
  processing it in multiple space-time scales. We demonstrate state-of-the-art
  text-to-video generation results, and show that our design easily facilitates a
  wide range of content creation tasks and video editing applications, including
  image-to-video, video inpainting, and stylized generation.
paper_authors._id: 65b06dc7fdf08903343bfeae
paper_authors.name: Omer Bar-Tal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:09:25.587Z
paper_authors._id: 62e29044a133a252b5cf70b2
paper_authors.avatarUrl: /avatars/6d09ddcba9bc47c309150a8d77815891.svg
paper_authors.fullname: Omer Bar-Tal
paper_authors.type: user
paper_authors.user: omerbartal
paper_authors._id: 65b06dc7fdf08903343bfeaf
paper_authors.name: Hila Chefer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-24T08:08:47.043Z
paper_authors._id: 6181c72cdcc1df2c9de8a4d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1655248010394-6181c72cdcc1df2c9de8a4d8.jpeg
paper_authors.fullname: Hila Chefer
paper_authors.type: user
paper_authors.user: Hila
paper_authors._id: 65b06dc7fdf08903343bfeb0
paper_authors.name: Omer Tov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:09:32.274Z
paper_authors._id: 623889093657dc674c30954e
paper_authors.avatarUrl: /avatars/a5a193d231e01448cc78ed5bba52bff5.svg
paper_authors.fullname: Omer Tov
paper_authors.type: user
paper_authors.user: omertov
paper_authors._id: 65b06dc7fdf08903343bfeb1
paper_authors.name: Charles Herrmann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:09:39.014Z
paper_authors._id: 64c7f088c864d962eddeaf8c
paper_authors.avatarUrl: /avatars/e6364c56e6cb6907b6dd2d34eed83720.svg
paper_authors.fullname: Charles Herrmann
paper_authors.type: user
paper_authors.user: irwinherrmann
paper_authors._id: 65b06dc7fdf08903343bfeb2
paper_authors.name: Roni Paiss
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:09:45.323Z
paper_authors._id: 62cab8aeb189b6164d26bfbe
paper_authors.avatarUrl: /avatars/7ed117e375dd9280a5750a242cae24ca.svg
paper_authors.fullname: Roni Paiss
paper_authors.type: user
paper_authors.user: RoniPaiss
paper_authors._id: 65b06dc7fdf08903343bfeb3
paper_authors.name: Shiran Zada
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:09:51.755Z
paper_authors._id: 63510f62cba4ff2e81cb0492
paper_authors.avatarUrl: /avatars/2375a841491de8f40f66b6d0fb0df7b1.svg
paper_authors.fullname: Shiran Zada
paper_authors.type: user
paper_authors.user: shiranzada
paper_authors._id: 65b06dc7fdf08903343bfeb4
paper_authors.name: Ariel Ephrat
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-29T08:26:18.807Z
paper_authors._id: 65b6ada5ff5235e6ca9551c9
paper_authors.avatarUrl: /avatars/2c558fd133bc1d0cd1b5d7f39d1968ee.svg
paper_authors.fullname: Ariel Ephrat
paper_authors.type: user
paper_authors.user: arielephrat
paper_authors._id: 65b06dc7fdf08903343bfeb5
paper_authors.name: Junhwa Hur
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:10:02.609Z
paper_authors._id: 647f3046b0e96764589e107a
paper_authors.avatarUrl: /avatars/1074ee5ec1b2d5d297406f9d455213c4.svg
paper_authors.fullname: Junhwa Hur
paper_authors.type: user
paper_authors.user: hurjunhwa
paper_authors._id: 65b06dc7fdf08903343bfeb6
paper_authors.name: Yuanzhen Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:10:09.523Z
paper_authors._id: 6334ee5831a2be3938c85639
paper_authors.avatarUrl: /avatars/9821af26a585f54b595ec39f061f2250.svg
paper_authors.fullname: Yuanzhen Li
paper_authors.type: user
paper_authors.user: yzli
paper_authors._id: 65b06dc7fdf08903343bfeb7
paper_authors.name: Tomer Michaeli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:10:16.729Z
paper_authors._id: 6457c3f50664b05a5b1d941d
paper_authors.avatarUrl: /avatars/0132be4d068d4b314f6b498b65411461.svg
paper_authors.fullname: Tomer Michaeli
paper_authors.type: user
paper_authors.user: Michato
paper_authors._id: 65b06dc7fdf08903343bfeb8
paper_authors.name: Oliver Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:11:23.096Z
paper_authors._id: 62fa8bfdc95d426ff8f25fe9
paper_authors.avatarUrl: /avatars/71fe9cff4ce79f6a208045984852e5ad.svg
paper_authors.fullname: Oliver Wang
paper_authors.type: user
paper_authors.user: oliverwang
paper_authors._id: 65b06dc7fdf08903343bfeb9
paper_authors.name: Deqing Sun
paper_authors._id: 65b06dc7fdf08903343bfeba
paper_authors.name: Tali Dekel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:11:37.971Z
paper_authors._id: 631cddec68f7da9ad24f6fc7
paper_authors.avatarUrl: /avatars/7d4f1ce805e5889ca6594bd4a93f2583.svg
paper_authors.fullname: Tali Dekel
paper_authors.type: user
paper_authors.user: talidekel
paper_authors._id: 65b06dc7fdf08903343bfebb
paper_authors.name: Inbar Mosseri
paper_title: Lumiere: A Space-Time Diffusion Model for Video Generation
paper_summary: We introduce Lumiere -- a text-to-video diffusion model designed for
  synthesizing videos that portray realistic, diverse and coherent motion -- a
  pivotal challenge in video synthesis. To this end, we introduce a Space-Time
  U-Net architecture that generates the entire temporal duration of the video at
  once, through a single pass in the model. This is in contrast to existing video
  models which synthesize distant keyframes followed by temporal super-resolution
  -- an approach that inherently makes global temporal consistency difficult to
  achieve. By deploying both spatial and (importantly) temporal down- and
  up-sampling and leveraging a pre-trained text-to-image diffusion model, our
  model learns to directly generate a full-frame-rate, low-resolution video by
  processing it in multiple space-time scales. We demonstrate state-of-the-art
  text-to-video generation results, and show that our design easily facilitates a
  wide range of content creation tasks and video editing applications, including
  image-to-video, video inpainting, and stylized generation.
paper_ai_summary: A text-to-video diffusion model using Space-Time U-Net architecture generates realistic, diverse, and coherent videos through a single pass, achieving state-of-the-art results and supporting various content creation and editing tasks.
paper_ai_keywords: Space-Time U-Net diffusion model temporal duration spatial down- and up-sampling temporal down- and up-sampling text-to-image diffusion model full-frame-rate low-resolution video image-to-video video inpainting stylized generation