title: MTV-Inpaint: Multi-Task Long Video Inpainting
summary: Video inpainting involves modifying local regions within a video, ensuring
  spatial and temporal consistency. Most existing methods focus primarily on
  scene completion (i.e., filling missing regions) and lack the capability to
  insert new objects into a scene in a controllable manner. Fortunately, recent
  advancements in text-to-video (T2V) diffusion models pave the way for
  text-guided video inpainting. However, directly adapting T2V models for
  inpainting remains limited in unifying completion and insertion tasks, lacks
  input controllability, and struggles with long videos, thereby restricting
  their applicability and flexibility. To address these challenges, we propose
  MTV-Inpaint, a unified multi-task video inpainting framework capable of
  handling both traditional scene completion and novel object insertion tasks. To
  unify these distinct tasks, we design a dual-branch spatial attention mechanism
  in the T2V diffusion U-Net, enabling seamless integration of scene completion
  and object insertion within a single framework. In addition to textual
  guidance, MTV-Inpaint supports multimodal control by integrating various image
  inpainting models through our proposed image-to-video (I2V) inpainting mode.
  Additionally, we propose a two-stage pipeline that combines keyframe inpainting
  with in-between frame propagation, enabling MTV-Inpaint to effectively handle
  long videos with hundreds of frames. Extensive experiments demonstrate that
  MTV-Inpaint achieves state-of-the-art performance in both scene completion and
  object insertion tasks. Furthermore, it demonstrates versatility in derived
  applications such as multi-modal inpainting, object editing, removal, image
  object brush, and the ability to handle long videos. Project page:
  https://mtv-inpaint.github.io/.
paper_authors._id: 67d8f8b77f61dda9ea6512b7
paper_authors.name: Shiyuan Yang
paper_authors._id: 67d8f8b77f61dda9ea6512b8
paper_authors.name: Zheng Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:33:49.096Z
paper_authors._id: 678f49878af7a399877b87c0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3IHnM4AKbIW_wmL15wdZf.png
paper_authors.fullname: GuZheng
paper_authors.type: user
paper_authors.user: GuZheng
paper_authors._id: 67d8f8b77f61dda9ea6512b9
paper_authors.name: Liang Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:33:57.501Z
paper_authors._id: 64560a2aaaaf85a98fa9a4b9
paper_authors.avatarUrl: /avatars/e81e21f353baf48f0d91bf29ad200eea.svg
paper_authors.fullname: Liang Hou
paper_authors.type: user
paper_authors.user: lianghou
paper_authors._id: 67d8f8b77f61dda9ea6512ba
paper_authors.name: Xin Tao
paper_authors._id: 67d8f8b77f61dda9ea6512bb
paper_authors.name: Pengfei Wan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-18T04:42:06.437Z
paper_authors._id: 662f93942510ef5735d7ad00
paper_authors.avatarUrl: /avatars/dc9486db75869ce902d0a638eea126bd.svg
paper_authors.fullname: magicwpf
paper_authors.type: user
paper_authors.user: magicwpf
paper_authors._id: 67d8f8b77f61dda9ea6512bc
paper_authors.name: Xiaodong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:34:15.858Z
paper_authors._id: 67be7cd21616162fc336cb44
paper_authors.avatarUrl: /avatars/e58cc3c2d1484419222a5ccfc11f5c48.svg
paper_authors.fullname: Xiaodong Chen
paper_authors.type: user
paper_authors.user: XiaodongChen
paper_authors._id: 67d8f8b77f61dda9ea6512bd
paper_authors.name: Jing Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:34:23.653Z
paper_authors._id: 65e77726767bfc7d109c45bf
paper_authors.avatarUrl: /avatars/24e68c86e06055ea1209598ba49ce8b9.svg
paper_authors.fullname: Jing Liao
paper_authors.type: user
paper_authors.user: CeciliaJL
paper_title: MTV-Inpaint: Multi-Task Long Video Inpainting
paper_summary: Video inpainting involves modifying local regions within a video, ensuring
  spatial and temporal consistency. Most existing methods focus primarily on
  scene completion (i.e., filling missing regions) and lack the capability to
  insert new objects into a scene in a controllable manner. Fortunately, recent
  advancements in text-to-video (T2V) diffusion models pave the way for
  text-guided video inpainting. However, directly adapting T2V models for
  inpainting remains limited in unifying completion and insertion tasks, lacks
  input controllability, and struggles with long videos, thereby restricting
  their applicability and flexibility. To address these challenges, we propose
  MTV-Inpaint, a unified multi-task video inpainting framework capable of
  handling both traditional scene completion and novel object insertion tasks. To
  unify these distinct tasks, we design a dual-branch spatial attention mechanism
  in the T2V diffusion U-Net, enabling seamless integration of scene completion
  and object insertion within a single framework. In addition to textual
  guidance, MTV-Inpaint supports multimodal control by integrating various image
  inpainting models through our proposed image-to-video (I2V) inpainting mode.
  Additionally, we propose a two-stage pipeline that combines keyframe inpainting
  with in-between frame propagation, enabling MTV-Inpaint to effectively handle
  long videos with hundreds of frames. Extensive experiments demonstrate that
  MTV-Inpaint achieves state-of-the-art performance in both scene completion and
  object insertion tasks. Furthermore, it demonstrates versatility in derived
  applications such as multi-modal inpainting, object editing, removal, image
  object brush, and the ability to handle long videos. Project page:
  https://mtv-inpaint.github.io/.
paper_ai_keywords: text-to-video (T2V) diffusion models text-guided video inpainting dual-branch spatial attention mechanism T2V diffusion U-Net multimodal control image-to-video (I2V) inpainting mode keyframe inpainting in-between frame propagation multi-modal inpainting object editing object removal image object brush