title: Hierarchical Masked 3D Diffusion Model for Video Outpainting
summary: Video outpainting aims to adequately complete missing areas at the edges of
  video frames. Compared to image outpainting, it presents an additional
  challenge as the model should maintain the temporal consistency of the filled
  area. In this paper, we introduce a masked 3D diffusion model for video
  outpainting. We use the technique of mask modeling to train the 3D diffusion
  model. This allows us to use multiple guide frames to connect the results of
  multiple video clip inferences, thus ensuring temporal consistency and reducing
  jitter between adjacent frames. Meanwhile, we extract the global frames of the
  video as prompts and guide the model to obtain information other than the
  current video clip using cross-attention. We also introduce a hybrid
  coarse-to-fine inference pipeline to alleviate the artifact accumulation
  problem. The existing coarse-to-fine pipeline only uses the infilling strategy,
  which brings degradation because the time interval of the sparse frames is too
  large. Our pipeline benefits from bidirectional learning of the mask modeling
  and thus can employ a hybrid strategy of infilling and interpolation when
  generating sparse frames. Experiments show that our method achieves
  state-of-the-art results in video outpainting tasks. More results are provided
  at our https://fanfanda.github.io/M3DDM/.
paper_authors._id: 64f830998a4c5c401d7d8aef
paper_authors.name: Fanda Fan
paper_authors._id: 64f830998a4c5c401d7d8af0
paper_authors.name: Chaoxu Guo
paper_authors._id: 64f830998a4c5c401d7d8af1
paper_authors.name: Litong Gong
paper_authors._id: 64f830998a4c5c401d7d8af2
paper_authors.name: Biao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-06T15:35:27.672Z
paper_authors._id: 64dcd9de0a0efd28e3b846e3
paper_authors.avatarUrl: /avatars/0732be326b1d3658118f669dd283d582.svg
paper_authors.fullname: ictwangbiao
paper_authors.type: user
paper_authors.user: ictwangbiao
paper_authors._id: 64f830998a4c5c401d7d8af3
paper_authors.name: Tiezheng Ge
paper_authors._id: 64f830998a4c5c401d7d8af4
paper_authors.name: Yuning Jiang
paper_authors._id: 64f830998a4c5c401d7d8af5
paper_authors.name: Chunjie Luo
paper_authors._id: 64f830998a4c5c401d7d8af6
paper_authors.name: Jianfeng Zhan
paper_title: Hierarchical Masked 3D Diffusion Model for Video Outpainting
paper_summary: Video outpainting aims to adequately complete missing areas at the edges of
  video frames. Compared to image outpainting, it presents an additional
  challenge as the model should maintain the temporal consistency of the filled
  area. In this paper, we introduce a masked 3D diffusion model for video
  outpainting. We use the technique of mask modeling to train the 3D diffusion
  model. This allows us to use multiple guide frames to connect the results of
  multiple video clip inferences, thus ensuring temporal consistency and reducing
  jitter between adjacent frames. Meanwhile, we extract the global frames of the
  video as prompts and guide the model to obtain information other than the
  current video clip using cross-attention. We also introduce a hybrid
  coarse-to-fine inference pipeline to alleviate the artifact accumulation
  problem. The existing coarse-to-fine pipeline only uses the infilling strategy,
  which brings degradation because the time interval of the sparse frames is too
  large. Our pipeline benefits from bidirectional learning of the mask modeling
  and thus can employ a hybrid strategy of infilling and interpolation when
  generating sparse frames. Experiments show that our method achieves
  state-of-the-art results in video outpainting tasks. More results are provided
  at our https://fanfanda.github.io/M3DDM/.
paper_ai_keywords: masked 3D diffusion model mask modeling guide frames temporal consistency global frames prompts cross-attention hybrid coarse-to-fine inference pipeline infilling strategy interpolation sparse frames bidirectional learning