title: Long-Context Autoregressive Video Modeling with Next-Frame Prediction
summary: Long-context autoregressive modeling has significantly advanced language
  generation, but video generation still struggles to fully utilize extended
  temporal contexts. To investigate long-context video modeling, we introduce
  Frame AutoRegressive (FAR), a strong baseline for video autoregressive
  modeling. Just as language models learn causal dependencies between tokens
  (i.e., Token AR), FAR models temporal causal dependencies between continuous
  frames, achieving better convergence than Token AR and video diffusion
  transformers. Building on FAR, we observe that long-context vision modeling
  faces challenges due to visual redundancy. Existing RoPE lacks effective
  temporal decay for remote context and fails to extrapolate well to long video
  sequences. Additionally, training on long videos is computationally expensive,
  as vision tokens grow much faster than language tokens. To tackle these issues,
  we propose balancing locality and long-range dependency. We introduce FlexRoPE,
  an test-time technique that adds flexible temporal decay to RoPE, enabling
  extrapolation to 16x longer vision contexts. Furthermore, we propose long
  short-term context modeling, where a high-resolution short-term context window
  ensures fine-grained temporal consistency, while an unlimited long-term context
  window encodes long-range information using fewer tokens. With this approach,
  we can train on long video sequences with a manageable token context length. We
  demonstrate that FAR achieves state-of-the-art performance in both short- and
  long-video generation, providing a simple yet effective baseline for video
  autoregressive modeling.
paper_authors._id: 67e35f6fc9d8214b5e1c64c3
paper_authors.name: Yuchao Gu
paper_authors._id: 67e35f6fc9d8214b5e1c64c4
paper_authors.name: Weijia Mao
paper_authors._id: 67e35f6fc9d8214b5e1c64c5
paper_authors.name: Mike Zheng Shou
paper_title: Long-Context Autoregressive Video Modeling with Next-Frame Prediction
paper_summary: Long-context autoregressive modeling has significantly advanced language
  generation, but video generation still struggles to fully utilize extended
  temporal contexts. To investigate long-context video modeling, we introduce
  Frame AutoRegressive (FAR), a strong baseline for video autoregressive
  modeling. Just as language models learn causal dependencies between tokens
  (i.e., Token AR), FAR models temporal causal dependencies between continuous
  frames, achieving better convergence than Token AR and video diffusion
  transformers. Building on FAR, we observe that long-context vision modeling
  faces challenges due to visual redundancy. Existing RoPE lacks effective
  temporal decay for remote context and fails to extrapolate well to long video
  sequences. Additionally, training on long videos is computationally expensive,
  as vision tokens grow much faster than language tokens. To tackle these issues,
  we propose balancing locality and long-range dependency. We introduce FlexRoPE,
  an test-time technique that adds flexible temporal decay to RoPE, enabling
  extrapolation to 16x longer vision contexts. Furthermore, we propose long
  short-term context modeling, where a high-resolution short-term context window
  ensures fine-grained temporal consistency, while an unlimited long-term context
  window encodes long-range information using fewer tokens. With this approach,
  we can train on long video sequences with a manageable token context length. We
  demonstrate that FAR achieves state-of-the-art performance in both short- and
  long-video generation, providing a simple yet effective baseline for video
  autoregressive modeling.
paper_ai_summary: FAR, an autoregressive video model, achieves state-of-the-art results by balancing locality and long-range context with FlexRoPE and long short-term context modeling, addressing challenges in long-context video generation.
paper_ai_keywords: Frame AutoRegressive (FAR) video autoregressive modeling causal dependencies Token AR video diffusion transformers long-context vision modeling visual redundancy RoPE temporal decay long video sequences FlexRoPE long short-term context modeling fine-grained temporal consistency long-range information short-video generation long-video generation