Taming Teacher Forcing for Masked Autoregressive Video Generation
We introduce MAGI, a hybrid video generation framework that combines masked
modeling for intra-frame generation with causal modeling for next-frame
generation. Our key innovation, Complete Teacher Forcing (CTF), conditions
masked frames on complete observation frames rather than masked ones (namely
Masked Teacher Forcing, MTF), enabling a smooth transition from token-level
(patch-level) to frame-level autoregressive generation. CTF significantly
outperforms MTF, achieving a +23% improvement in FVD scores on first-frame
conditioned video prediction. To address issues like exposure bias, we employ
targeted training strategies, setting a new benchmark in autoregressive video
generation. Experiments show that MAGI can generate long, coherent video
sequences exceeding 100 frames, even when trained on as few as 16 frames,
highlighting its potential for scalable, high-quality video generation.
6791564a65b96bef4a168e00
Deyu Zhou
admin_assigned
2025-01-22T20:54:25.055Z
64e821f2bddc5b1072b15c2e
/avatars/618b5a48f2fa62daff4e1922a9aa9e8b.svg
zhoudeyu
user
zhoudeyu
6791564a65b96bef4a168e01
Quan Sun
admin_assigned
2025-01-22T20:54:39.129Z
630d7a8f81ef9b1772b67f4c
/avatars/00757abd6e548ccebb5bfb233be129a2.svg
Quan Sun
user
QuanSun
6791564a65b96bef4a168e02
Yuang Peng
admin_assigned
2025-01-22T20:54:44.656Z
631ee086c1a8269da39265c6
https://cdn-avatars.huggingface.co/v1/production/uploads/631ee086c1a8269da39265c6/wUa1epGtTGcUv2mvrLUcD.png
Yuang Peng
user
yuangpeng
6791564a65b96bef4a168e03
Kun Yan
6791564a65b96bef4a168e04
Runpei Dong
claimed_verified
2025-01-22T20:49:49.933Z
6201fc5d91d53938a6432fbf
https://cdn-avatars.huggingface.co/v1/production/uploads/6201fc5d91d53938a6432fbf/VLs8ZYaZrop4KBpZn53fH.jpeg
Runpei Dong
user
RunpeiDong
6791564a65b96bef4a168e05
Duomin Wang
claimed_verified
2025-01-23T09:18:05.586Z
64ae9b88a22a179fc4d07992
/avatars/c9065f04a1188ea3129e56a90328ffd3.svg
wang
user
dorni
6791564a65b96bef4a168e06
Zheng Ge
6791564a65b96bef4a168e07
Nan Duan
admin_assigned
2025-01-22T20:55:16.663Z
66422aede3e9eba09d3cb753
/avatars/52a6292fcc3a37265062bdcfbea73441.svg
Nan Duan
user
opotle
6791564a65b96bef4a168e08
Xiangyu Zhang
6791564a65b96bef4a168e09
Lionel M. Ni
6791564a65b96bef4a168e0a
Heung-Yeung Shum
Taming Teacher Forcing for Masked Autoregressive Video Generation
We introduce MAGI, a hybrid video generation framework that combines masked
modeling for intra-frame generation with causal modeling for next-frame
generation. Our key innovation, Complete Teacher Forcing (CTF), conditions
masked frames on complete observation frames rather than masked ones (namely
Masked Teacher Forcing, MTF), enabling a smooth transition from token-level
(patch-level) to frame-level autoregressive generation. CTF significantly
outperforms MTF, achieving a +23% improvement in FVD scores on first-frame
conditioned video prediction. To address issues like exposure bias, we employ
targeted training strategies, setting a new benchmark in autoregressive video
generation. Experiments show that MAGI can generate long, coherent video
sequences exceeding 100 frames, even when trained on as few as 16 frames,
highlighting its potential for scalable, high-quality video generation.
MAGI, a hybrid video generation framework using masked and causal modeling, achieves superior performance with Complete Teacher Forcing, overcoming exposure bias and generating long, coherent video sequences.
masked modeling intra-frame generation causal modeling next-frame generation Complete Teacher Forcing Masked Teacher Forcing FVD scores first-frame conditioned video prediction autoregressive generation exposure bias targeted training strategies video prediction video generation