title: MarDini: Masked Autoregressive Diffusion for Video Generation at Scale
summary: We introduce MarDini, a new family of video diffusion models that integrate
  the advantages of masked auto-regression (MAR) into a unified diffusion model
  (DM) framework. Here, MAR handles temporal planning, while DM focuses on
  spatial generation in an asymmetric network design: i) a MAR-based planning
  model containing most of the parameters generates planning signals for each
  masked frame using low-resolution input; ii) a lightweight generation model
  uses these signals to produce high-resolution frames via diffusion de-noising.
  MarDini's MAR enables video generation conditioned on any number of masked
  frames at any frame positions: a single model can handle video interpolation
  (e.g., masking middle frames), image-to-video generation (e.g., masking from
  the second frame onward), and video expansion (e.g., masking half the frames).
  The efficient design allocates most of the computational resources to the
  low-resolution planning model, making computationally expensive but important
  spatio-temporal attention feasible at scale. MarDini sets a new
  state-of-the-art for video interpolation; meanwhile, within few inference
  steps, it efficiently generates videos on par with those of much more expensive
  advanced image-to-video models.
paper_authors._id: 6720583f93f3d8192f15db37
paper_authors.name: Haozhe Liu
paper_authors._id: 6720583f93f3d8192f15db38
paper_authors.name: Shikun Liu
paper_authors._id: 6720583f93f3d8192f15db39
paper_authors.name: Zijian Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:54:14.379Z
paper_authors._id: 62e0fcfe4db2175cd2710063
paper_authors.avatarUrl: /avatars/0f8d1b5ba3e3eba2af18f98f65f0b180.svg
paper_authors.fullname: Zijian Zhou
paper_authors.type: user
paper_authors.user: franciszzj
paper_authors._id: 6720583f93f3d8192f15db3a
paper_authors.name: Mengmeng Xu
paper_authors._id: 6720583f93f3d8192f15db3b
paper_authors.name: Yanping Xie
paper_authors._id: 6720583f93f3d8192f15db3c
paper_authors.name: Xiao Han
paper_authors._id: 6720583f93f3d8192f15db3d
paper_authors.name: Juan C. Pérez
paper_authors._id: 6720583f93f3d8192f15db3e
paper_authors.name: Ding Liu
paper_authors._id: 6720583f93f3d8192f15db3f
paper_authors.name: Kumara Kahatapitiya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:54:16.358Z
paper_authors._id: 65a12ec4c9873890d15c4ab9
paper_authors.avatarUrl: /avatars/ce4d46f575ba757f78eabdb25b394171.svg
paper_authors.fullname: Kumara Kahatapitiya
paper_authors.type: user
paper_authors.user: kumarak
paper_authors._id: 6720583f93f3d8192f15db40
paper_authors.name: Menglin Jia
paper_authors._id: 6720583f93f3d8192f15db41
paper_authors.name: Jui-Chieh Wu
paper_authors._id: 6720583f93f3d8192f15db42
paper_authors.name: Sen He
paper_authors._id: 6720583f93f3d8192f15db43
paper_authors.name: Tao Xiang
paper_authors._id: 6720583f93f3d8192f15db44
paper_authors.name: Jürgen Schmidhuber
paper_authors._id: 6720583f93f3d8192f15db45
paper_authors.name: Juan-Manuel Pérez-Rúa
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:54:12.675Z
paper_authors._id: 6572f4d86a1344d777fa7535
paper_authors.avatarUrl: /avatars/eb19a0d4e8d8019456a78a7d621d0955.svg
paper_authors.fullname: Juanma Perez
paper_authors.type: user
paper_authors.user: jperezrua
paper_title: MarDini: Masked Autoregressive Diffusion for Video Generation at Scale
paper_summary: We introduce MarDini, a new family of video diffusion models that integrate
  the advantages of masked auto-regression (MAR) into a unified diffusion model
  (DM) framework. Here, MAR handles temporal planning, while DM focuses on
  spatial generation in an asymmetric network design: i) a MAR-based planning
  model containing most of the parameters generates planning signals for each
  masked frame using low-resolution input; ii) a lightweight generation model
  uses these signals to produce high-resolution frames via diffusion de-noising.
  MarDini's MAR enables video generation conditioned on any number of masked
  frames at any frame positions: a single model can handle video interpolation
  (e.g., masking middle frames), image-to-video generation (e.g., masking from
  the second frame onward), and video expansion (e.g., masking half the frames).
  The efficient design allocates most of the computational resources to the
  low-resolution planning model, making computationally expensive but important
  spatio-temporal attention feasible at scale. MarDini sets a new
  state-of-the-art for video interpolation; meanwhile, within few inference
  steps, it efficiently generates videos on par with those of much more expensive
  advanced image-to-video models.
paper_ai_keywords: video diffusion models masked auto-regression (MAR) diffusion model (DM) network design MAR-based planning model generation model diffusion de-noising video interpolation image-to-video generation video expansion computational resources spatio-temporal attention state-of-the-art inference steps