title: StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation
    from Text
summary: Text-to-video diffusion models enable the generation of high-quality videos
  that follow text instructions, making it easy to create diverse and individual
  content. However, existing approaches mostly focus on high-quality short video
  generation (typically 16 or 24 frames), ending up with hard-cuts when naively
  extended to the case of long video synthesis. To overcome these limitations, we
  introduce StreamingT2V, an autoregressive approach for long video generation of
  80, 240, 600, 1200 or more frames with smooth transitions. The key components
  are:(i) a short-term memory block called conditional attention module (CAM),
  which conditions the current generation on the features extracted from the
  previous chunk via an attentional mechanism, leading to consistent chunk
  transitions, (ii) a long-term memory block called appearance preservation
  module, which extracts high-level scene and object features from the first
  video chunk to prevent the model from forgetting the initial scene, and (iii) a
  randomized blending approach that enables to apply a video enhancer
  autoregressively for infinitely long videos without inconsistencies between
  chunks. Experiments show that StreamingT2V generates high motion amount. In
  contrast, all competing image-to-video methods are prone to video stagnation
  when applied naively in an autoregressive manner. Thus, we propose with
  StreamingT2V a high-quality seamless text-to-long video generator that
  outperforms competitors with consistency and motion. Our code will be available
  at: https://github.com/Picsart-AI-Research/StreamingT2V
paper_authors._id: 6600d2748270811586f8e8cf
paper_authors.name: Roberto Henschel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:18:38.071Z
paper_authors._id: 6316145f95b55e2621d0a128
paper_authors.avatarUrl: /avatars/6a35dbd6f48b6e34a6fc8b8d04e00f6a.svg
paper_authors.fullname: Roberto Henschel
paper_authors.type: user
paper_authors.user: robhen
paper_authors._id: 6600d2748270811586f8e8d0
paper_authors.name: Levon Khachatryan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:19:22.407Z
paper_authors._id: 6380b78b5c62156ce7e08c3d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6380b78b5c62156ce7e08c3d/qCc7j9uCMRYowbYJYwASy.jpeg
paper_authors.fullname: Levon Khachatryan
paper_authors.type: user
paper_authors.user: lev1
paper_authors._id: 6600d2748270811586f8e8d1
paper_authors.name: Daniil Hayrapetyan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:19:28.359Z
paper_authors._id: 6315a76d86b3db2ac4203bcc
paper_authors.avatarUrl: /avatars/46094173beafd663e29990874058d121.svg
paper_authors.fullname: Daniil Hayrapetyan
paper_authors.type: user
paper_authors.user: oldnaari
paper_authors._id: 6600d2748270811586f8e8d2
paper_authors.name: Hayk Poghosyan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:19:34.305Z
paper_authors._id: 64e1a70c75fae2212e9336b9
paper_authors.avatarUrl: /avatars/97ba153584d10d69ee33b71c310b04a0.svg
paper_authors.fullname: Hayk Poghosyan
paper_authors.type: user
paper_authors.user: haykpoghos
paper_authors._id: 6600d2748270811586f8e8d3
paper_authors.name: Vahram Tadevosyan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:19:40.383Z
paper_authors._id: 63e0cb125c6964861ebb27ac
paper_authors.avatarUrl: /avatars/c22663168e7b9d0a9fc70bc9c5fc7f02.svg
paper_authors.fullname: Vahram Tadevosyan
paper_authors.type: user
paper_authors.user: vahramtadevosyan
paper_authors._id: 6600d2748270811586f8e8d4
paper_authors.name: Zhangyang Wang
paper_authors._id: 6600d2748270811586f8e8d5
paper_authors.name: Shant Navasardyan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:20:05.842Z
paper_authors._id: 641dadf816ef0e5a7275a5cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641dadf816ef0e5a7275a5cd/YrieHABBK9VH82VxYPJnp.jpeg
paper_authors.fullname: Shant Navasardyan
paper_authors.type: user
paper_authors.user: navasardyanshant
paper_authors._id: 6600d2748270811586f8e8d6
paper_authors.name: Humphrey Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:20:12.061Z
paper_authors._id: 61e1188afc27c0f5e3641eb3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e1188afc27c0f5e3641eb3/R4JFlWgYomZFvZ9CLSZfD.jpeg
paper_authors.fullname: Humphrey Shi
paper_authors.type: user
paper_authors.user: Humphrey
paper_title: StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation
    from Text
paper_summary: Text-to-video diffusion models enable the generation of high-quality videos
  that follow text instructions, making it easy to create diverse and individual
  content. However, existing approaches mostly focus on high-quality short video
  generation (typically 16 or 24 frames), ending up with hard-cuts when naively
  extended to the case of long video synthesis. To overcome these limitations, we
  introduce StreamingT2V, an autoregressive approach for long video generation of
  80, 240, 600, 1200 or more frames with smooth transitions. The key components
  are:(i) a short-term memory block called conditional attention module (CAM),
  which conditions the current generation on the features extracted from the
  previous chunk via an attentional mechanism, leading to consistent chunk
  transitions, (ii) a long-term memory block called appearance preservation
  module, which extracts high-level scene and object features from the first
  video chunk to prevent the model from forgetting the initial scene, and (iii) a
  randomized blending approach that enables to apply a video enhancer
  autoregressively for infinitely long videos without inconsistencies between
  chunks. Experiments show that StreamingT2V generates high motion amount. In
  contrast, all competing image-to-video methods are prone to video stagnation
  when applied naively in an autoregressive manner. Thus, we propose with
  StreamingT2V a high-quality seamless text-to-long video generator that
  outperforms competitors with consistency and motion. Our code will be available
  at: https://github.com/Picsart-AI-Research/StreamingT2V
paper_ai_summary: StreamingT2V is an autoregressive text-to-video diffusion model that generates long, high-quality videos with smooth transitions by incorporating short-term and long-term memory blocks and a randomized blending approach.
paper_ai_keywords: text-to-video diffusion models autoregressive approach conditional attention module (CAM) appearance preservation module randomized blending video enhancer video stagnation high motion amount