title: LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation
    with Linear Computational Complexity
summary: Text-to-video generation enhances content creation but is highly
  computationally intensive: The computational cost of Diffusion Transformers
  (DiTs) scales quadratically in the number of pixels. This makes minute-length
  video generation extremely expensive, limiting most existing models to
  generating videos of only 10-20 seconds length. We propose a Linear-complexity
  text-to-video Generation (LinGen) framework whose cost scales linearly in the
  number of pixels. For the first time, LinGen enables high-resolution
  minute-length video generation on a single GPU without compromising quality. It
  replaces the computationally-dominant and quadratic-complexity block,
  self-attention, with a linear-complexity block called MATE, which consists of
  an MA-branch and a TE-branch. The MA-branch targets short-to-long-range
  correlations, combining a bidirectional Mamba2 block with our token
  rearrangement method, Rotary Major Scan, and our review tokens developed for
  long video generation. The TE-branch is a novel TEmporal Swin Attention block
  that focuses on temporal correlations between adjacent tokens and medium-range
  tokens. The MATE block addresses the adjacency preservation issue of Mamba and
  improves the consistency of generated videos significantly. Experimental
  results show that LinGen outperforms DiT (with a 75.6% win rate) in video
  quality with up to 15times (11.5times) FLOPs (latency) reduction.
  Furthermore, both automatic metrics and human evaluation demonstrate our
  LinGen-4B yields comparable video quality to state-of-the-art models (with a
  50.5%, 52.1%, 49.1% win rate with respect to Gen-3, LumaLabs, and Kling,
  respectively). This paves the way to hour-length movie generation and real-time
  interactive video generation. We provide 68s video generation results and more
  examples in our project website: https://lineargen.github.io/.
paper_authors._id: 675f9f0bcd95ed0ab888fa28
paper_authors.name: Hongjie Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:16:27.354Z
paper_authors._id: 63c8d37c9cf77a4393d4e16c
paper_authors.avatarUrl: /avatars/58a44bdad26718c3814fcdfa589aed1e.svg
paper_authors.fullname: Hongjie Wang
paper_authors.type: user
paper_authors.user: hongjiewang
paper_authors._id: 675f9f0bcd95ed0ab888fa29
paper_authors.name: Chih-Yao Ma
paper_authors._id: 675f9f0bcd95ed0ab888fa2a
paper_authors.name: Yen-Cheng Liu
paper_authors._id: 675f9f0bcd95ed0ab888fa2b
paper_authors.name: Ji Hou
paper_authors._id: 675f9f0bcd95ed0ab888fa2c
paper_authors.name: Tao Xu
paper_authors._id: 675f9f0bcd95ed0ab888fa2d
paper_authors.name: Jialiang Wang
paper_authors._id: 675f9f0bcd95ed0ab888fa2e
paper_authors.name: Felix Juefei-Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T16:55:17.348Z
paper_authors._id: 6417cf37dce1e4c0229f17b1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6417cf37dce1e4c0229f17b1/7h-ZCB5f4wif7TsnF-B1M.jpeg
paper_authors.fullname: Felix Xu
paper_authors.type: user
paper_authors.user: katanaxu
paper_authors._id: 675f9f0bcd95ed0ab888fa2f
paper_authors.name: Yaqiao Luo
paper_authors._id: 675f9f0bcd95ed0ab888fa30
paper_authors.name: Peizhao Zhang
paper_authors._id: 675f9f0bcd95ed0ab888fa31
paper_authors.name: Tingbo Hou
paper_authors._id: 675f9f0bcd95ed0ab888fa32
paper_authors.name: Peter Vajda
paper_authors._id: 675f9f0bcd95ed0ab888fa33
paper_authors.name: Niraj K. Jha
paper_authors._id: 675f9f0bcd95ed0ab888fa34
paper_authors.name: Xiaoliang Dai
paper_title: LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation
    with Linear Computational Complexity
paper_summary: Text-to-video generation enhances content creation but is highly
  computationally intensive: The computational cost of Diffusion Transformers
  (DiTs) scales quadratically in the number of pixels. This makes minute-length
  video generation extremely expensive, limiting most existing models to
  generating videos of only 10-20 seconds length. We propose a Linear-complexity
  text-to-video Generation (LinGen) framework whose cost scales linearly in the
  number of pixels. For the first time, LinGen enables high-resolution
  minute-length video generation on a single GPU without compromising quality. It
  replaces the computationally-dominant and quadratic-complexity block,
  self-attention, with a linear-complexity block called MATE, which consists of
  an MA-branch and a TE-branch. The MA-branch targets short-to-long-range
  correlations, combining a bidirectional Mamba2 block with our token
  rearrangement method, Rotary Major Scan, and our review tokens developed for
  long video generation. The TE-branch is a novel TEmporal Swin Attention block
  that focuses on temporal correlations between adjacent tokens and medium-range
  tokens. The MATE block addresses the adjacency preservation issue of Mamba and
  improves the consistency of generated videos significantly. Experimental
  results show that LinGen outperforms DiT (with a 75.6% win rate) in video
  quality with up to 15times (11.5times) FLOPs (latency) reduction.
  Furthermore, both automatic metrics and human evaluation demonstrate our
  LinGen-4B yields comparable video quality to state-of-the-art models (with a
  50.5%, 52.1%, 49.1% win rate with respect to Gen-3, LumaLabs, and Kling,
  respectively). This paves the way to hour-length movie generation and real-time
  interactive video generation. We provide 68s video generation results and more
  examples in our project website: https://lineargen.github.io/.
paper_ai_keywords: Diffusion Transformers (DiTs) Linear-complexity text-to-video Generation (LinGen) self-attention MATE MA-branch TE-branch Mamba2 block Rotary Major Scan review tokens TEmporal Swin Attention block