StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation
  from Text
Text-to-video diffusion models enable the generation of high-quality videos
that follow text instructions, making it easy to create diverse and individual
content. However, existing approaches mostly focus on high-quality short video
generation (typically 16 or 24 frames), ending up with hard-cuts when naively
extended to the case of long video synthesis. To overcome these limitations, we
introduce StreamingT2V, an autoregressive approach for long video generation of
80, 240, 600, 1200 or more frames with smooth transitions. The key components
are:(i) a short-term memory block called conditional attention module (CAM),
which conditions the current generation on the features extracted from the
previous chunk via an attentional mechanism, leading to consistent chunk
transitions, (ii) a long-term memory block called appearance preservation
module, which extracts high-level scene and object features from the first
video chunk to prevent the model from forgetting the initial scene, and (iii) a
randomized blending approach that enables to apply a video enhancer
autoregressively for infinitely long videos without inconsistencies between
chunks. Experiments show that StreamingT2V generates high motion amount. In
contrast, all competing image-to-video methods are prone to video stagnation
when applied naively in an autoregressive manner. Thus, we propose with
StreamingT2V a high-quality seamless text-to-long video generator that
outperforms competitors with consistency and motion. Our code will be available
at: https://github.com/Picsart-AI-Research/StreamingT2V
6600d2748270811586f8e8cf
Roberto Henschel
admin_assigned
2024-03-25T10:18:38.071Z
6316145f95b55e2621d0a128
/avatars/6a35dbd6f48b6e34a6fc8b8d04e00f6a.svg
Roberto Henschel
user
robhen
6600d2748270811586f8e8d0
Levon Khachatryan
admin_assigned
2024-03-25T10:19:22.407Z
6380b78b5c62156ce7e08c3d
https://cdn-avatars.huggingface.co/v1/production/uploads/6380b78b5c62156ce7e08c3d/qCc7j9uCMRYowbYJYwASy.jpeg
Levon Khachatryan
user
lev1
6600d2748270811586f8e8d1
Daniil Hayrapetyan
admin_assigned
2024-03-25T10:19:28.359Z
6315a76d86b3db2ac4203bcc
/avatars/46094173beafd663e29990874058d121.svg
Daniil Hayrapetyan
user
oldnaari
6600d2748270811586f8e8d2
Hayk Poghosyan
admin_assigned
2024-03-25T10:19:34.305Z
64e1a70c75fae2212e9336b9
/avatars/97ba153584d10d69ee33b71c310b04a0.svg
Hayk Poghosyan
user
haykpoghos
6600d2748270811586f8e8d3
Vahram Tadevosyan
admin_assigned
2024-03-25T10:19:40.383Z
63e0cb125c6964861ebb27ac
/avatars/c22663168e7b9d0a9fc70bc9c5fc7f02.svg
Vahram Tadevosyan
user
vahramtadevosyan
6600d2748270811586f8e8d4
Zhangyang Wang
6600d2748270811586f8e8d5
Shant Navasardyan
admin_assigned
2024-03-25T10:20:05.842Z
641dadf816ef0e5a7275a5cd
https://cdn-avatars.huggingface.co/v1/production/uploads/641dadf816ef0e5a7275a5cd/YrieHABBK9VH82VxYPJnp.jpeg
Shant Navasardyan
user
navasardyanshant
6600d2748270811586f8e8d6
Humphrey Shi
admin_assigned
2024-03-25T10:20:12.061Z
61e1188afc27c0f5e3641eb3
https://cdn-avatars.huggingface.co/v1/production/uploads/61e1188afc27c0f5e3641eb3/R4JFlWgYomZFvZ9CLSZfD.jpeg
Humphrey Shi
user
Humphrey
StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation
  from Text
Text-to-video diffusion models enable the generation of high-quality videos
that follow text instructions, making it easy to create diverse and individual
content. However, existing approaches mostly focus on high-quality short video
generation (typically 16 or 24 frames), ending up with hard-cuts when naively
extended to the case of long video synthesis. To overcome these limitations, we
introduce StreamingT2V, an autoregressive approach for long video generation of
80, 240, 600, 1200 or more frames with smooth transitions. The key components
are:(i) a short-term memory block called conditional attention module (CAM),
which conditions the current generation on the features extracted from the
previous chunk via an attentional mechanism, leading to consistent chunk
transitions, (ii) a long-term memory block called appearance preservation
module, which extracts high-level scene and object features from the first
video chunk to prevent the model from forgetting the initial scene, and (iii) a
randomized blending approach that enables to apply a video enhancer
autoregressively for infinitely long videos without inconsistencies between
chunks. Experiments show that StreamingT2V generates high motion amount. In
contrast, all competing image-to-video methods are prone to video stagnation
when applied naively in an autoregressive manner. Thus, we propose with
StreamingT2V a high-quality seamless text-to-long video generator that
outperforms competitors with consistency and motion. Our code will be available
at: https://github.com/Picsart-AI-Research/StreamingT2V
StreamingT2V is an autoregressive text-to-video diffusion model that generates long, high-quality videos with smooth transitions by incorporating short-term and long-term memory blocks and a randomized blending approach.
text-to-video diffusion models autoregressive approach conditional attention module (CAM) appearance preservation module randomized blending video enhancer video stagnation high motion amount