Autoregressive Video Generation without Vector Quantization
This paper presents a novel approach that enables autoregressive video
generation with high efficiency. We propose to reformulate the video generation
problem as a non-quantized autoregressive modeling of temporal frame-by-frame
prediction and spatial set-by-set prediction. Unlike raster-scan prediction in
prior autoregressive models or joint distribution modeling of fixed-length
tokens in diffusion models, our approach maintains the causal property of
GPT-style models for flexible in-context capabilities, while leveraging
bidirectional modeling within individual frames for efficiency. With the
proposed approach, we train a novel video autoregressive model without vector
quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior
autoregressive video models in data efficiency, inference speed, visual
fidelity, and video fluency, even with a much smaller model capacity, i.e.,
0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models
in text-to-image generation tasks, with a significantly lower training cost.
Additionally, NOVA generalizes well across extended video durations and enables
diverse zero-shot applications in one unified model. Code and models are
publicly available at https://github.com/baaivision/NOVA.
6764675e0dffe9a6526186ae
Haoge Deng
claimed_verified
2024-12-20T08:33:28.506Z
64dd9404f2c8f66d52604fb3
/avatars/1d7fec043e0848a145e4dcaf8973a575.svg
Haoge Deng
user
Bitterdhg
6764675e0dffe9a6526186af
Ting Pan
claimed_verified
2024-12-21T15:19:56.321Z
6565bc5ee5aac326bfc98e39
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vIfHy9Y1yAK6A96UCHNBH.jpeg
Ting Pan
user
PhyscalX
6764675e0dffe9a6526186b0
Haiwen Diao
admin_assigned
2024-12-20T09:59:41.879Z
64b4a717aa03b6520839e9b8
https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
Haiwen Diao
user
Paranioar
6764675e0dffe9a6526186b1
Zhengxiong Luo
claimed_verified
2024-12-20T10:08:50.623Z
64afbb93bbdbfc1b8921960b
/avatars/9ed45fa8a051c10cddfe2c9a3624d619.svg
zhengxiong
user
lzxlog
6764675e0dffe9a6526186b2
Yufeng Cui
admin_assigned
2024-12-20T09:59:27.648Z
648683de623b5f050213f2be
/avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
Yufeng Cui
user
YufengCui
6764675e0dffe9a6526186b3
Huchuan Lu
6764675e0dffe9a6526186b4
Shiguang Shan
6764675e0dffe9a6526186b5
Yonggang Qi
6764675e0dffe9a6526186b6
Xinlong Wang
admin_assigned
2024-12-20T09:58:57.601Z
63ca558304c979828311c5a5
/avatars/2a439d79fba2f987cabe780d10c94d25.svg
Xinlong Wang
user
xinlongwang
Autoregressive Video Generation without Vector Quantization
This paper presents a novel approach that enables autoregressive video
generation with high efficiency. We propose to reformulate the video generation
problem as a non-quantized autoregressive modeling of temporal frame-by-frame
prediction and spatial set-by-set prediction. Unlike raster-scan prediction in
prior autoregressive models or joint distribution modeling of fixed-length
tokens in diffusion models, our approach maintains the causal property of
GPT-style models for flexible in-context capabilities, while leveraging
bidirectional modeling within individual frames for efficiency. With the
proposed approach, we train a novel video autoregressive model without vector
quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior
autoregressive video models in data efficiency, inference speed, visual
fidelity, and video fluency, even with a much smaller model capacity, i.e.,
0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models
in text-to-image generation tasks, with a significantly lower training cost.
Additionally, NOVA generalizes well across extended video durations and enables
diverse zero-shot applications in one unified model. Code and models are
publicly available at https://github.com/baaivision/NOVA.
A new autoregressive video generation model, NOVA, achieves high efficiency, surpassing prior models and diffusion methods in various tasks with fewer parameters.
autoregressive modeling temporal frame-by-frame prediction spatial set-by-set prediction raster-scan prediction GPT-style models bidirectional modeling vector quantization NOVA data efficiency inference speed visual fidelity video fluency text-to-image generation