title: Lumina-Video: Efficient and Flexible Video Generation with Multi-scale
    Next-DiT
summary: Recent advancements have established Diffusion Transformers (DiTs) as a
  dominant framework in generative modeling. Building on this success,
  Lumina-Next achieves exceptional performance in the generation of
  photorealistic images with Next-DiT. However, its potential for video
  generation remains largely untapped, with significant challenges in modeling
  the spatiotemporal complexity inherent to video data. To address this, we
  introduce Lumina-Video, a framework that leverages the strengths of Next-DiT
  while introducing tailored solutions for video synthesis. Lumina-Video
  incorporates a Multi-scale Next-DiT architecture, which jointly learns multiple
  patchifications to enhance both efficiency and flexibility. By incorporating
  the motion score as an explicit condition, Lumina-Video also enables direct
  control of generated videos' dynamic degree. Combined with a progressive
  training scheme with increasingly higher resolution and FPS, and a multi-source
  training scheme with mixed natural and synthetic data, Lumina-Video achieves
  remarkable aesthetic quality and motion smoothness at high training and
  inference efficiency. We additionally propose Lumina-V2A, a video-to-audio
  model based on Next-DiT, to create synchronized sounds for generated videos.
  Codes are released at https://www.github.com/Alpha-VLLM/Lumina-Video.
paper_authors._id: 67aae76c71a9983f50e134ef
paper_authors.name: Dongyang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-13T19:27:55.452Z
paper_authors._id: 646f1bef075e11ca78da3bb7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646f1bef075e11ca78da3bb7/gNS-ikyZXYeMrf4a7HTQE.jpeg
paper_authors.fullname: Dongyang Liu (Chris Liu)
paper_authors.type: user
paper_authors.user: Cxxs
paper_authors._id: 67aae76c71a9983f50e134f0
paper_authors.name: Shicheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T12:42:28.636Z
paper_authors._id: 64539588fc2b5f69e8faac76
paper_authors.avatarUrl: /avatars/111441eeb0dd4d8ad2f0d3f28277952a.svg
paper_authors.fullname: Li Shicheng
paper_authors.type: user
paper_authors.user: lscpku
paper_authors._id: 67aae76c71a9983f50e134f1
paper_authors.name: Yutong Liu
paper_authors._id: 67aae76c71a9983f50e134f2
paper_authors.name: Zhen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T15:37:44.523Z
paper_authors._id: 6285a9133ab6642179158944
paper_authors.avatarUrl: /avatars/6e10fa07c94141fcdbe0cab02bb731ca.svg
paper_authors.fullname: Zhen Li
paper_authors.type: user
paper_authors.user: Paper99
paper_authors._id: 67aae76c71a9983f50e134f3
paper_authors.name: Kai Wang
paper_authors._id: 67aae76c71a9983f50e134f4
paper_authors.name: Xinyue Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-28T08:38:44.243Z
paper_authors._id: 66aba287b0f0b7411f511a47
paper_authors.avatarUrl: /avatars/1450f182c38e80066ae5ea5df4fa218f.svg
paper_authors.fullname: Xinyue Li
paper_authors.type: user
paper_authors.user: Xxxy13
paper_authors._id: 67aae76c71a9983f50e134f5
paper_authors.name: Qi Qin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-28T08:38:46.150Z
paper_authors._id: 66bb136002fd8eb58bc84ffb
paper_authors.avatarUrl: /avatars/122cb8f59c502392768099b3c2afe043.svg
paper_authors.fullname: qinqi
paper_authors.type: user
paper_authors.user: Dakerqi
paper_authors._id: 67aae76c71a9983f50e134f6
paper_authors.name: Yufei Liu
paper_authors._id: 67aae76c71a9983f50e134f7
paper_authors.name: Yi Xin
paper_authors._id: 67aae76c71a9983f50e134f8
paper_authors.name: Zhongyu Li
paper_authors._id: 67aae76c71a9983f50e134f9
paper_authors.name: Bin Fu
paper_authors._id: 67aae76c71a9983f50e134fa
paper_authors.name: Chenyang Si
paper_authors._id: 67aae76c71a9983f50e134fb
paper_authors.name: Yuewen Cao
paper_authors._id: 67aae76c71a9983f50e134fc
paper_authors.name: Conghui He
paper_authors._id: 67aae76c71a9983f50e134fd
paper_authors.name: Ziwei Liu
paper_authors._id: 67aae76c71a9983f50e134fe
paper_authors.name: Yu Qiao
paper_authors._id: 67aae76c71a9983f50e134ff
paper_authors.name: Qibin Hou
paper_authors._id: 67aae76c71a9983f50e13500
paper_authors.name: Hongsheng Li
paper_authors._id: 67aae76c71a9983f50e13501
paper_authors.name: Peng Gao
paper_title: Lumina-Video: Efficient and Flexible Video Generation with Multi-scale
    Next-DiT
paper_summary: Recent advancements have established Diffusion Transformers (DiTs) as a
  dominant framework in generative modeling. Building on this success,
  Lumina-Next achieves exceptional performance in the generation of
  photorealistic images with Next-DiT. However, its potential for video
  generation remains largely untapped, with significant challenges in modeling
  the spatiotemporal complexity inherent to video data. To address this, we
  introduce Lumina-Video, a framework that leverages the strengths of Next-DiT
  while introducing tailored solutions for video synthesis. Lumina-Video
  incorporates a Multi-scale Next-DiT architecture, which jointly learns multiple
  patchifications to enhance both efficiency and flexibility. By incorporating
  the motion score as an explicit condition, Lumina-Video also enables direct
  control of generated videos' dynamic degree. Combined with a progressive
  training scheme with increasingly higher resolution and FPS, and a multi-source
  training scheme with mixed natural and synthetic data, Lumina-Video achieves
  remarkable aesthetic quality and motion smoothness at high training and
  inference efficiency. We additionally propose Lumina-V2A, a video-to-audio
  model based on Next-DiT, to create synchronized sounds for generated videos.
  Codes are released at https://www.github.com/Alpha-VLLM/Lumina-Video.
paper_ai_summary: Lumina-Video enhances video generation by combining Multi-scale Next-DiT with motion scoring and multi-source training, achieving high quality and efficiency.
paper_ai_keywords: Diffusion Transformers DiTs Next-DiT Lumina-Next Lumina-Video Multi-scale Next-DiT patchifications motion score progressive training FPS multi-source training Lumina-V2A video-to-audio model