LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation
  with Linear Computational Complexity
Text-to-video generation enhances content creation but is highly
computationally intensive: The computational cost of Diffusion Transformers
(DiTs) scales quadratically in the number of pixels. This makes minute-length
video generation extremely expensive, limiting most existing models to
generating videos of only 10-20 seconds length. We propose a Linear-complexity
text-to-video Generation (LinGen) framework whose cost scales linearly in the
number of pixels. For the first time, LinGen enables high-resolution
minute-length video generation on a single GPU without compromising quality. It
replaces the computationally-dominant and quadratic-complexity block,
self-attention, with a linear-complexity block called MATE, which consists of
an MA-branch and a TE-branch. The MA-branch targets short-to-long-range
correlations, combining a bidirectional Mamba2 block with our token
rearrangement method, Rotary Major Scan, and our review tokens developed for
long video generation. The TE-branch is a novel TEmporal Swin Attention block
that focuses on temporal correlations between adjacent tokens and medium-range
tokens. The MATE block addresses the adjacency preservation issue of Mamba and
improves the consistency of generated videos significantly. Experimental
results show that LinGen outperforms DiT (with a 75.6% win rate) in video
quality with up to 15times (11.5times) FLOPs (latency) reduction.
Furthermore, both automatic metrics and human evaluation demonstrate our
LinGen-4B yields comparable video quality to state-of-the-art models (with a
50.5%, 52.1%, 49.1% win rate with respect to Gen-3, LumaLabs, and Kling,
respectively). This paves the way to hour-length movie generation and real-time
interactive video generation. We provide 68s video generation results and more
examples in our project website: https://lineargen.github.io/.
675f9f0bcd95ed0ab888fa28
Hongjie Wang
claimed_verified
2024-12-16T09:16:27.354Z
63c8d37c9cf77a4393d4e16c
/avatars/58a44bdad26718c3814fcdfa589aed1e.svg
Hongjie Wang
user
hongjiewang
675f9f0bcd95ed0ab888fa29
Chih-Yao Ma
675f9f0bcd95ed0ab888fa2a
Yen-Cheng Liu
675f9f0bcd95ed0ab888fa2b
Ji Hou
claimed_verified
2025-04-13T19:28:12.373Z
63084636f0dc38fb47c2dc6b
/avatars/04edff10a048b5d93981e6f5af3ecf40.svg
Ji Hou
user
jhou90
675f9f0bcd95ed0ab888fa2c
Tao Xu
675f9f0bcd95ed0ab888fa2d
Jialiang Wang
675f9f0bcd95ed0ab888fa2e
Felix Juefei-Xu
claimed_verified
2024-12-16T16:55:17.348Z
6417cf37dce1e4c0229f17b1
https://cdn-avatars.huggingface.co/v1/production/uploads/6417cf37dce1e4c0229f17b1/7h-ZCB5f4wif7TsnF-B1M.jpeg
Felix Xu
user
katanaxu
675f9f0bcd95ed0ab888fa2f
Yaqiao Luo
675f9f0bcd95ed0ab888fa30
Peizhao Zhang
675f9f0bcd95ed0ab888fa31
Tingbo Hou
675f9f0bcd95ed0ab888fa32
Peter Vajda
675f9f0bcd95ed0ab888fa33
Niraj K. Jha
675f9f0bcd95ed0ab888fa34
Xiaoliang Dai
LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation
  with Linear Computational Complexity
Text-to-video generation enhances content creation but is highly
computationally intensive: The computational cost of Diffusion Transformers
(DiTs) scales quadratically in the number of pixels. This makes minute-length
video generation extremely expensive, limiting most existing models to
generating videos of only 10-20 seconds length. We propose a Linear-complexity
text-to-video Generation (LinGen) framework whose cost scales linearly in the
number of pixels. For the first time, LinGen enables high-resolution
minute-length video generation on a single GPU without compromising quality. It
replaces the computationally-dominant and quadratic-complexity block,
self-attention, with a linear-complexity block called MATE, which consists of
an MA-branch and a TE-branch. The MA-branch targets short-to-long-range
correlations, combining a bidirectional Mamba2 block with our token
rearrangement method, Rotary Major Scan, and our review tokens developed for
long video generation. The TE-branch is a novel TEmporal Swin Attention block
that focuses on temporal correlations between adjacent tokens and medium-range
tokens. The MATE block addresses the adjacency preservation issue of Mamba and
improves the consistency of generated videos significantly. Experimental
results show that LinGen outperforms DiT (with a 75.6% win rate) in video
quality with up to 15times (11.5times) FLOPs (latency) reduction.
Furthermore, both automatic metrics and human evaluation demonstrate our
LinGen-4B yields comparable video quality to state-of-the-art models (with a
50.5%, 52.1%, 49.1% win rate with respect to Gen-3, LumaLabs, and Kling,
respectively). This paves the way to hour-length movie generation and real-time
interactive video generation. We provide 68s video generation results and more
examples in our project website: https://lineargen.github.io/.
LinGen, a text-to-video generation framework using a linear-complexity MATE block, enables high-quality, high-resolution minute-length video generation on a single GPU with significantly reduced computational cost.
Diffusion Transformers text-to-video Generation Linear-complexity self-attention MATE MA-branch TE-branch Mamba2 Rotary Major Scan review tokens TEmporal Swin Attention adjacency preservation