Next Block Prediction: Video Generation via Semi-Autoregressive Modeling
Next-Token Prediction (NTP) is a de facto approach for autoregressive (AR)
video generation, but it suffers from suboptimal unidirectional dependencies
and slow inference speed. In this work, we propose a semi-autoregressive
(semi-AR) framework, called Next-Block Prediction (NBP), for video generation.
By uniformly decomposing video content into equal-sized blocks (e.g., rows or
frames), we shift the generation unit from individual tokens to blocks,
allowing each token in the current block to simultaneously predict the
corresponding token in the next block. Unlike traditional AR modeling, our
framework employs bidirectional attention within each block, enabling tokens to
capture more robust spatial dependencies. By predicting multiple tokens in
parallel, NBP models significantly reduce the number of generation steps,
leading to faster and more efficient inference. Our model achieves FVD scores
of 103.3 on UCF101 and 25.5 on K600, outperforming the vanilla NTP model by an
average of 4.4. Furthermore, thanks to the reduced number of inference steps,
the NBP model generates 8.89 frames (128x128 resolution) per second, achieving
an 11x speedup. We also explored model scales ranging from 700M to 3B
parameters, observing significant improvements in generation quality, with FVD
scores dropping from 103.3 to 55.3 on UCF101 and from 25.5 to 19.5 on K600,
demonstrating the scalability of our approach.
67ad5d2f8436e8ea7abb7a15
Shuhuai Ren
claimed_verified
2025-03-24T08:55:45.971Z
60d2e681b8448e1785bbda06
https://cdn-avatars.huggingface.co/v1/production/uploads/1624434302056-noauth.jpeg
Shuhuai Ren
user
ShuhuaiRen
67ad5d2f8436e8ea7abb7a16
Shuming Ma
67ad5d2f8436e8ea7abb7a17
Xu Sun
67ad5d2f8436e8ea7abb7a18
Furu Wei
extracted_pending
2025-04-02T06:16:49.324Z
67ecd6178647cfa1775f75ed
/avatars/98882cc58dc0a5de94df765d523d92c9.svg
FW
user
frontierai
Next Block Prediction: Video Generation via Semi-Autoregressive Modeling
Next-Token Prediction (NTP) is a de facto approach for autoregressive (AR)
video generation, but it suffers from suboptimal unidirectional dependencies
and slow inference speed. In this work, we propose a semi-autoregressive
(semi-AR) framework, called Next-Block Prediction (NBP), for video generation.
By uniformly decomposing video content into equal-sized blocks (e.g., rows or
frames), we shift the generation unit from individual tokens to blocks,
allowing each token in the current block to simultaneously predict the
corresponding token in the next block. Unlike traditional AR modeling, our
framework employs bidirectional attention within each block, enabling tokens to
capture more robust spatial dependencies. By predicting multiple tokens in
parallel, NBP models significantly reduce the number of generation steps,
leading to faster and more efficient inference. Our model achieves FVD scores
of 103.3 on UCF101 and 25.5 on K600, outperforming the vanilla NTP model by an
average of 4.4. Furthermore, thanks to the reduced number of inference steps,
the NBP model generates 8.89 frames (128x128 resolution) per second, achieving
an 11x speedup. We also explored model scales ranging from 700M to 3B
parameters, observing significant improvements in generation quality, with FVD
scores dropping from 103.3 to 55.3 on UCF101 and from 25.5 to 19.5 on K600,
demonstrating the scalability of our approach.
A semi-autoregressive video generation framework, Next-Block Prediction (NBP), improves inference speed and quality by predicting video blocks in parallel with bidirectional attention.
Next-Token Prediction autoregressive semi-autoregressive Next-Block Prediction bidirectional attention FVD UCF101 K600 video generation