Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video
  Synthesis
Contemporary models for generating images show remarkable quality and
versatility. Swayed by these advantages, the research community repurposes them
to generate videos. Since video content is highly redundant, we argue that
naively bringing advances of image models to the video generation domain
reduces motion fidelity, visual quality and impairs scalability. In this work,
we build Snap Video, a video-first model that systematically addresses these
challenges. To do that, we first extend the EDM framework to take into account
spatially and temporally redundant pixels and naturally support video
generation. Second, we show that a U-Net - a workhorse behind image generation
- scales poorly when generating videos, requiring significant computational
overhead. Hence, we propose a new transformer-based architecture that trains
3.31 times faster than U-Nets (and is ~4.5 faster at inference). This allows us
to efficiently train a text-to-video model with billions of parameters for the
first time, reach state-of-the-art results on a number of benchmarks, and
generate videos with substantially higher quality, temporal consistency, and
motion complexity. The user studies showed that our model was favored by a
large margin over the most recent methods. See our website at
https://snap-research.github.io/snapvideo/.
65d80312685624d5f2dfed19
Willi Menapace
65d80312685624d5f2dfed1a
Aliaksandr Siarohin
admin_assigned
2024-02-23T12:31:47.387Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
65d80312685624d5f2dfed1b
Ivan Skorokhodov
admin_assigned
2024-02-23T12:31:54.000Z
63610db13c7147fae7de88e3
/avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
Ivan Skorokhodov
user
universome
65d80312685624d5f2dfed1c
Ekaterina Deyneka
admin_assigned
2024-02-23T12:32:00.729Z
635764d4d11d85992d00e014
/avatars/f4234ec5c56ac956545643751c5db9aa.svg
Ekaterina Deyneka
user
edeyneka
65d80312685624d5f2dfed1d
Tsai-Shien Chen
admin_assigned
2024-02-23T12:32:06.809Z
6330f570b68c7453d2ee8945
/avatars/5b394a72f277aa6c886b21b477010ab0.svg
Tsai-Shien Chen
user
tschen
65d80312685624d5f2dfed1e
Anil Kag
65d80312685624d5f2dfed1f
Yuwei Fang
65d80312685624d5f2dfed20
Aleksei Stoliar
65d80312685624d5f2dfed21
Elisa Ricci
65d80312685624d5f2dfed22
Jian Ren
claimed_verified
2024-02-24T14:37:00.550Z
61f19829233c91cbd2f79e70
/avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
Jian Ren
user
alanspike
65d80312685624d5f2dfed23
Sergey Tulyakov
Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video
  Synthesis
Contemporary models for generating images show remarkable quality and
versatility. Swayed by these advantages, the research community repurposes them
to generate videos. Since video content is highly redundant, we argue that
naively bringing advances of image models to the video generation domain
reduces motion fidelity, visual quality and impairs scalability. In this work,
we build Snap Video, a video-first model that systematically addresses these
challenges. To do that, we first extend the EDM framework to take into account
spatially and temporally redundant pixels and naturally support video
generation. Second, we show that a U-Net - a workhorse behind image generation
- scales poorly when generating videos, requiring significant computational
overhead. Hence, we propose a new transformer-based architecture that trains
3.31 times faster than U-Nets (and is ~4.5 faster at inference). This allows us
to efficiently train a text-to-video model with billions of parameters for the
first time, reach state-of-the-art results on a number of benchmarks, and
generate videos with substantially higher quality, temporal consistency, and
motion complexity. The user studies showed that our model was favored by a
large margin over the most recent methods. See our website at
https://snap-research.github.io/snapvideo/.
Snap Video, a transformer-based model, addresses video generation challenges by extending EDM for spatial and temporal redundancy, achieving superior quality, consistency, and speed compared to existing U-Net based methods.
EDM framework U-Net transformer-based architecture text-to-video model motion fidelity visual quality temporal consistency motion complexity