Lumos-1: On Autoregressive Video Generation from a Unified Model
  Perspective
Autoregressive large language models (LLMs) have unified a vast range of
language tasks, inspiring preliminary efforts in autoregressive video
generation. Existing autoregressive video generators either diverge from
standard LLM architectures, depend on bulky external text encoders, or incur
prohibitive latency due to next-token decoding. In this paper, we introduce
Lumos-1, an autoregressive video generator that retains the LLM architecture
with minimal architectural modifications. To inject spatiotemporal correlations
in LLMs, we identify the efficacy of incorporating 3D RoPE and diagnose its
imbalanced frequency spectrum ranges. Therefore, we propose MM-RoPE, a RoPE
scheme that preserves the original textual RoPE while providing comprehensive
frequency spectra and scaled 3D positions for modeling multimodal
spatiotemporal data. Moreover, Lumos-1 resorts to a token dependency strategy
that obeys intra-frame bidirectionality and inter-frame temporal causality.
Based on this dependency strategy, we identify the issue of frame-wise loss
imbalance caused by spatial information redundancy and solve it by proposing
Autoregressive Discrete Diffusion Forcing (AR-DF). AR-DF introduces temporal
tube masking during training with a compatible inference-time masking policy to
avoid quality degradation. By using memory-efficient training techniques, we
pre-train Lumos-1 on only 48 GPUs, achieving performance comparable to EMU3 on
GenEval, COSMOS-Video2World on VBench-I2V, and OpenSoraPlan on VBench-T2V. Code
and models are available at https://github.com/alibaba-damo-academy/Lumos.
68746fc3257d4f04353702ce
Hangjie Yuan
claimed_verified
2025-07-15T19:10:41.143Z
649d54b314afbb10ce2a9eeb
/avatars/15c325d8c2273ff63569f23015e98486.svg
Hangjie Yuan
user
JacobYuan
68746fc3257d4f04353702cf
Weihua Chen
68746fc3257d4f04353702d0
Jun Cen
claimed_verified
2025-07-15T19:10:39.038Z
644493bb9c1bd83bd1a09860
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VNb6fnFYraR5R4vSwMYZz.jpeg
Jun CEN
user
jcenaa
68746fc3257d4f04353702d1
Hu Yu
68746fc3257d4f04353702d2
Jingyun Liang
68746fc3257d4f04353702d3
Shuning Chang
68746fc3257d4f04353702d4
Zhihui Lin
68746fc3257d4f04353702d5
Tao Feng
68746fc3257d4f04353702d6
Pengwei Liu
68746fc3257d4f04353702d7
Jiazheng Xing
68746fc3257d4f04353702d8
Hao Luo
68746fc3257d4f04353702d9
Jiasheng Tang
68746fc3257d4f04353702da
Fan Wang
68746fc3257d4f04353702db
Yi Yang
Lumos-1: On Autoregressive Video Generation from a Unified Model
  Perspective
Autoregressive large language models (LLMs) have unified a vast range of
language tasks, inspiring preliminary efforts in autoregressive video
generation. Existing autoregressive video generators either diverge from
standard LLM architectures, depend on bulky external text encoders, or incur
prohibitive latency due to next-token decoding. In this paper, we introduce
Lumos-1, an autoregressive video generator that retains the LLM architecture
with minimal architectural modifications. To inject spatiotemporal correlations
in LLMs, we identify the efficacy of incorporating 3D RoPE and diagnose its
imbalanced frequency spectrum ranges. Therefore, we propose MM-RoPE, a RoPE
scheme that preserves the original textual RoPE while providing comprehensive
frequency spectra and scaled 3D positions for modeling multimodal
spatiotemporal data. Moreover, Lumos-1 resorts to a token dependency strategy
that obeys intra-frame bidirectionality and inter-frame temporal causality.
Based on this dependency strategy, we identify the issue of frame-wise loss
imbalance caused by spatial information redundancy and solve it by proposing
Autoregressive Discrete Diffusion Forcing (AR-DF). AR-DF introduces temporal
tube masking during training with a compatible inference-time masking policy to
avoid quality degradation. By using memory-efficient training techniques, we
pre-train Lumos-1 on only 48 GPUs, achieving performance comparable to EMU3 on
GenEval, COSMOS-Video2World on VBench-I2V, and OpenSoraPlan on VBench-T2V. Code
and models are available at https://github.com/alibaba-damo-academy/Lumos.
Lumos-1 is an autoregressive video generator that uses modified LLM architecture with MM-RoPE and AR-DF to handle spatiotemporal data efficiently, achieving competitive performance with reduced computational resources.
autoregressive large language models LLMs autoregressive video generation 3D RoPE MM-RoPE token dependency strategy intra-frame bidirectionality inter-frame temporal causality frame-wise loss imbalance Autoregressive Discrete Diffusion Forcing AR-DF temporal tube masking memory-efficient training GenEval COSMOS-Video2World VBench-I2V OpenSoraPlan VBench-T2V