MarDini: Masked Autoregressive Diffusion for Video Generation at Scale
We introduce MarDini, a new family of video diffusion models that integrate
the advantages of masked auto-regression (MAR) into a unified diffusion model
(DM) framework. Here, MAR handles temporal planning, while DM focuses on
spatial generation in an asymmetric network design: i) a MAR-based planning
model containing most of the parameters generates planning signals for each
masked frame using low-resolution input; ii) a lightweight generation model
uses these signals to produce high-resolution frames via diffusion de-noising.
MarDini's MAR enables video generation conditioned on any number of masked
frames at any frame positions: a single model can handle video interpolation
(e.g., masking middle frames), image-to-video generation (e.g., masking from
the second frame onward), and video expansion (e.g., masking half the frames).
The efficient design allocates most of the computational resources to the
low-resolution planning model, making computationally expensive but important
spatio-temporal attention feasible at scale. MarDini sets a new
state-of-the-art for video interpolation; meanwhile, within few inference
steps, it efficiently generates videos on par with those of much more expensive
advanced image-to-video models.
6720583f93f3d8192f15db37
Haozhe Liu
6720583f93f3d8192f15db38
Shikun Liu
6720583f93f3d8192f15db39
Zijian Zhou
claimed_verified
2024-10-30T09:54:14.379Z
62e0fcfe4db2175cd2710063
/avatars/0f8d1b5ba3e3eba2af18f98f65f0b180.svg
Zijian Zhou
user
franciszzj
6720583f93f3d8192f15db3a
Mengmeng Xu
6720583f93f3d8192f15db3b
Yanping Xie
6720583f93f3d8192f15db3c
Xiao Han
6720583f93f3d8192f15db3d
Juan C. Pérez
6720583f93f3d8192f15db3e
Ding Liu
6720583f93f3d8192f15db3f
Kumara Kahatapitiya
claimed_verified
2024-10-30T09:54:16.358Z
65a12ec4c9873890d15c4ab9
/avatars/ce4d46f575ba757f78eabdb25b394171.svg
Kumara Kahatapitiya
user
kumarak
6720583f93f3d8192f15db40
Menglin Jia
6720583f93f3d8192f15db41
Jui-Chieh Wu
6720583f93f3d8192f15db42
Sen He
6720583f93f3d8192f15db43
Tao Xiang
6720583f93f3d8192f15db44
Jürgen Schmidhuber
6720583f93f3d8192f15db45
Juan-Manuel Pérez-Rúa
claimed_verified
2024-10-30T09:54:12.675Z
6572f4d86a1344d777fa7535
/avatars/eb19a0d4e8d8019456a78a7d621d0955.svg
Juanma Perez
user
jperezrua
MarDini: Masked Autoregressive Diffusion for Video Generation at Scale
We introduce MarDini, a new family of video diffusion models that integrate
the advantages of masked auto-regression (MAR) into a unified diffusion model
(DM) framework. Here, MAR handles temporal planning, while DM focuses on
spatial generation in an asymmetric network design: i) a MAR-based planning
model containing most of the parameters generates planning signals for each
masked frame using low-resolution input; ii) a lightweight generation model
uses these signals to produce high-resolution frames via diffusion de-noising.
MarDini's MAR enables video generation conditioned on any number of masked
frames at any frame positions: a single model can handle video interpolation
(e.g., masking middle frames), image-to-video generation (e.g., masking from
the second frame onward), and video expansion (e.g., masking half the frames).
The efficient design allocates most of the computational resources to the
low-resolution planning model, making computationally expensive but important
spatio-temporal attention feasible at scale. MarDini sets a new
state-of-the-art for video interpolation; meanwhile, within few inference
steps, it efficiently generates videos on par with those of much more expensive
advanced image-to-video models.
MarDini combines masked auto-regression and diffusion modeling for efficient video generation, accommodating various scenarios like interpolation and expansion with high quality.
masked auto-regression diffusion models asymmetric network design planning model generation model diffusion de-noising spatio-temporal attention