title: Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video
    Synthesis
summary: Contemporary models for generating images show remarkable quality and
  versatility. Swayed by these advantages, the research community repurposes them
  to generate videos. Since video content is highly redundant, we argue that
  naively bringing advances of image models to the video generation domain
  reduces motion fidelity, visual quality and impairs scalability. In this work,
  we build Snap Video, a video-first model that systematically addresses these
  challenges. To do that, we first extend the EDM framework to take into account
  spatially and temporally redundant pixels and naturally support video
  generation. Second, we show that a U-Net - a workhorse behind image generation
  - scales poorly when generating videos, requiring significant computational
  overhead. Hence, we propose a new transformer-based architecture that trains
  3.31 times faster than U-Nets (and is ~4.5 faster at inference). This allows us
  to efficiently train a text-to-video model with billions of parameters for the
  first time, reach state-of-the-art results on a number of benchmarks, and
  generate videos with substantially higher quality, temporal consistency, and
  motion complexity. The user studies showed that our model was favored by a
  large margin over the most recent methods. See our website at
  https://snap-research.github.io/snapvideo/.
paper_authors._id: 65d80312685624d5f2dfed19
paper_authors.name: Willi Menapace
paper_authors._id: 65d80312685624d5f2dfed1a
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T12:31:47.387Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 65d80312685624d5f2dfed1b
paper_authors.name: Ivan Skorokhodov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T12:31:54.000Z
paper_authors._id: 63610db13c7147fae7de88e3
paper_authors.avatarUrl: /avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
paper_authors.fullname: Ivan Skorokhodov
paper_authors.type: user
paper_authors.user: universome
paper_authors._id: 65d80312685624d5f2dfed1c
paper_authors.name: Ekaterina Deyneka
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T12:32:00.729Z
paper_authors._id: 635764d4d11d85992d00e014
paper_authors.avatarUrl: /avatars/f4234ec5c56ac956545643751c5db9aa.svg
paper_authors.fullname: Ekaterina Deyneka
paper_authors.type: user
paper_authors.user: edeyneka
paper_authors._id: 65d80312685624d5f2dfed1d
paper_authors.name: Tsai-Shien Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T12:32:06.809Z
paper_authors._id: 6330f570b68c7453d2ee8945
paper_authors.avatarUrl: /avatars/5b394a72f277aa6c886b21b477010ab0.svg
paper_authors.fullname: Tsai-Shien Chen
paper_authors.type: user
paper_authors.user: tschen
paper_authors._id: 65d80312685624d5f2dfed1e
paper_authors.name: Anil Kag
paper_authors._id: 65d80312685624d5f2dfed1f
paper_authors.name: Yuwei Fang
paper_authors._id: 65d80312685624d5f2dfed20
paper_authors.name: Aleksei Stoliar
paper_authors._id: 65d80312685624d5f2dfed21
paper_authors.name: Elisa Ricci
paper_authors._id: 65d80312685624d5f2dfed22
paper_authors.name: Jian Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-24T14:37:00.550Z
paper_authors._id: 61f19829233c91cbd2f79e70
paper_authors.avatarUrl: /avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
paper_authors.fullname: Jian Ren
paper_authors.type: user
paper_authors.user: alanspike
paper_authors._id: 65d80312685624d5f2dfed23
paper_authors.name: Sergey Tulyakov
paper_title: Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video
    Synthesis
paper_summary: Contemporary models for generating images show remarkable quality and
  versatility. Swayed by these advantages, the research community repurposes them
  to generate videos. Since video content is highly redundant, we argue that
  naively bringing advances of image models to the video generation domain
  reduces motion fidelity, visual quality and impairs scalability. In this work,
  we build Snap Video, a video-first model that systematically addresses these
  challenges. To do that, we first extend the EDM framework to take into account
  spatially and temporally redundant pixels and naturally support video
  generation. Second, we show that a U-Net - a workhorse behind image generation
  - scales poorly when generating videos, requiring significant computational
  overhead. Hence, we propose a new transformer-based architecture that trains
  3.31 times faster than U-Nets (and is ~4.5 faster at inference). This allows us
  to efficiently train a text-to-video model with billions of parameters for the
  first time, reach state-of-the-art results on a number of benchmarks, and
  generate videos with substantially higher quality, temporal consistency, and
  motion complexity. The user studies showed that our model was favored by a
  large margin over the most recent methods. See our website at
  https://snap-research.github.io/snapvideo/.
paper_ai_keywords: EDM framework spatially and temporally redundant pixels U-Net transformer-based architecture parameter-efficient text-to-video model state-of-the-art temporal consistency motion complexity