title: Autoregressive Video Generation without Vector Quantization
summary: This paper presents a novel approach that enables autoregressive video
  generation with high efficiency. We propose to reformulate the video generation
  problem as a non-quantized autoregressive modeling of temporal frame-by-frame
  prediction and spatial set-by-set prediction. Unlike raster-scan prediction in
  prior autoregressive models or joint distribution modeling of fixed-length
  tokens in diffusion models, our approach maintains the causal property of
  GPT-style models for flexible in-context capabilities, while leveraging
  bidirectional modeling within individual frames for efficiency. With the
  proposed approach, we train a novel video autoregressive model without vector
  quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior
  autoregressive video models in data efficiency, inference speed, visual
  fidelity, and video fluency, even with a much smaller model capacity, i.e.,
  0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models
  in text-to-image generation tasks, with a significantly lower training cost.
  Additionally, NOVA generalizes well across extended video durations and enables
  diverse zero-shot applications in one unified model. Code and models are
  publicly available at https://github.com/baaivision/NOVA.
paper_authors._id: 6764675e0dffe9a6526186ae
paper_authors.name: Haoge Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:33:28.506Z
paper_authors._id: 64dd9404f2c8f66d52604fb3
paper_authors.avatarUrl: /avatars/1d7fec043e0848a145e4dcaf8973a575.svg
paper_authors.fullname: Haoge Deng
paper_authors.type: user
paper_authors.user: Bitterdhg
paper_authors._id: 6764675e0dffe9a6526186af
paper_authors.name: Ting Pan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-21T15:19:56.321Z
paper_authors._id: 6565bc5ee5aac326bfc98e39
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vIfHy9Y1yAK6A96UCHNBH.jpeg
paper_authors.fullname: Ting Pan
paper_authors.type: user
paper_authors.user: PhyscalX
paper_authors._id: 6764675e0dffe9a6526186b0
paper_authors.name: Haiwen Diao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:59:41.879Z
paper_authors._id: 64b4a717aa03b6520839e9b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
paper_authors.fullname: Haiwen Diao
paper_authors.type: user
paper_authors.user: Paranioar
paper_authors._id: 6764675e0dffe9a6526186b1
paper_authors.name: Zhengxiong Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T10:08:50.623Z
paper_authors._id: 64afbb93bbdbfc1b8921960b
paper_authors.avatarUrl: /avatars/9ed45fa8a051c10cddfe2c9a3624d619.svg
paper_authors.fullname: zhengxiong
paper_authors.type: user
paper_authors.user: lzxlog
paper_authors._id: 6764675e0dffe9a6526186b2
paper_authors.name: Yufeng Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:59:27.648Z
paper_authors._id: 648683de623b5f050213f2be
paper_authors.avatarUrl: /avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
paper_authors.fullname: Yufeng Cui
paper_authors.type: user
paper_authors.user: YufengCui
paper_authors._id: 6764675e0dffe9a6526186b3
paper_authors.name: Huchuan Lu
paper_authors._id: 6764675e0dffe9a6526186b4
paper_authors.name: Shiguang Shan
paper_authors._id: 6764675e0dffe9a6526186b5
paper_authors.name: Yonggang Qi
paper_authors._id: 6764675e0dffe9a6526186b6
paper_authors.name: Xinlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:58:57.601Z
paper_authors._id: 63ca558304c979828311c5a5
paper_authors.avatarUrl: /avatars/2a439d79fba2f987cabe780d10c94d25.svg
paper_authors.fullname: Xinlong Wang
paper_authors.type: user
paper_authors.user: xinlongwang
paper_title: Autoregressive Video Generation without Vector Quantization
paper_summary: This paper presents a novel approach that enables autoregressive video
  generation with high efficiency. We propose to reformulate the video generation
  problem as a non-quantized autoregressive modeling of temporal frame-by-frame
  prediction and spatial set-by-set prediction. Unlike raster-scan prediction in
  prior autoregressive models or joint distribution modeling of fixed-length
  tokens in diffusion models, our approach maintains the causal property of
  GPT-style models for flexible in-context capabilities, while leveraging
  bidirectional modeling within individual frames for efficiency. With the
  proposed approach, we train a novel video autoregressive model without vector
  quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior
  autoregressive video models in data efficiency, inference speed, visual
  fidelity, and video fluency, even with a much smaller model capacity, i.e.,
  0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models
  in text-to-image generation tasks, with a significantly lower training cost.
  Additionally, NOVA generalizes well across extended video durations and enables
  diverse zero-shot applications in one unified model. Code and models are
  publicly available at https://github.com/baaivision/NOVA.
paper_ai_keywords: autoregressive video generation frame-by-frame prediction set-by-set prediction raster-scan prediction non-quant quantized aut diffusion models GPT-style models causal property bidirectional modeling NOVA character efficiency inference speed visual fidelity video fluency model capacity text-to-image generation state-of-the-art image diffusion models