title: VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play
    Context Control
summary: Video inpainting, which aims to restore corrupted video content, has
  experienced substantial progress. Despite these advances, existing methods,
  whether propagating unmasked region pixels through optical flow and receptive
  field priors, or extending image-inpainting models temporally, face challenges
  in generating fully masked objects or balancing the competing objectives of
  background context preservation and foreground generation in one model,
  respectively. To address these limitations, we propose a novel dual-stream
  paradigm VideoPainter that incorporates an efficient context encoder
  (comprising only 6% of the backbone parameters) to process masked videos and
  inject backbone-aware background contextual cues to any pre-trained video DiT,
  producing semantically consistent content in a plug-and-play manner. This
  architectural separation significantly reduces the model's learning complexity
  while enabling nuanced integration of crucial background context. We also
  introduce a novel target region ID resampling technique that enables any-length
  video inpainting, greatly enhancing our practical applicability. Additionally,
  we establish a scalable dataset pipeline leveraging current vision
  understanding models, contributing VPData and VPBench to facilitate
  segmentation-based inpainting training and assessment, the largest video
  inpainting dataset and benchmark to date with over 390K diverse clips. Using
  inpainting as a pipeline basis, we also explore downstream applications
  including video editing and video editing pair data generation, demonstrating
  competitive performance and significant practical potential. Extensive
  experiments demonstrate VideoPainter's superior performance in both any-length
  video inpainting and editing, across eight key metrics, including video
  quality, mask region preservation, and textual coherence.
paper_authors._id: 67ce5ad85847e4787a82242d
paper_authors.name: Yuxuan Bian
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-10T03:22:04.947Z
paper_authors._id: 650447dd52ca06fef957f05d
paper_authors.avatarUrl: /avatars/511c11ac9b3cc7a162bda5e07f6ee0a3.svg
paper_authors.fullname: Yuxuan BIAN
paper_authors.type: user
paper_authors.user: BianYx
paper_authors._id: 67ce5ad85847e4787a82242e
paper_authors.name: Zhaoyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:23:16.437Z
paper_authors._id: 658409ceca19ccf6d9989add
paper_authors.avatarUrl: /avatars/3ac1dbd25e52435185babdeb3da28875.svg
paper_authors.fullname: Zhaoyang Zhang
paper_authors.type: user
paper_authors.user: ZyZcuhk
paper_authors._id: 67ce5ad85847e4787a82242f
paper_authors.name: Xuan Ju
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:57:06.955Z
paper_authors._id: 62d4577bc85b0fcf7fde39bb
paper_authors.avatarUrl: /avatars/a3a5729e33ae89ce9ba408830db3c835.svg
paper_authors.fullname: Xuan Ju
paper_authors.type: user
paper_authors.user: juxuan27
paper_authors._id: 67ce5ad85847e4787a822430
paper_authors.name: Mingdeng Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:56:45.412Z
paper_authors._id: 6374a02d0856ac905bfc6113
paper_authors.avatarUrl: /avatars/2cbe75c9cc818a647ca6e416f129c96f.svg
paper_authors.fullname: Mingdeng Cao
paper_authors.type: user
paper_authors.user: Ljzycmd
paper_authors._id: 67ce5ad85847e4787a822431
paper_authors.name: Liangbin Xie
paper_authors._id: 67ce5ad85847e4787a822432
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:56:34.001Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_authors._id: 67ce5ad85847e4787a822433
paper_authors.name: Qiang Xu
paper_title: VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play
    Context Control
paper_summary: Video inpainting, which aims to restore corrupted video content, has
  experienced substantial progress. Despite these advances, existing methods,
  whether propagating unmasked region pixels through optical flow and receptive
  field priors, or extending image-inpainting models temporally, face challenges
  in generating fully masked objects or balancing the competing objectives of
  background context preservation and foreground generation in one model,
  respectively. To address these limitations, we propose a novel dual-stream
  paradigm VideoPainter that incorporates an efficient context encoder
  (comprising only 6% of the backbone parameters) to process masked videos and
  inject backbone-aware background contextual cues to any pre-trained video DiT,
  producing semantically consistent content in a plug-and-play manner. This
  architectural separation significantly reduces the model's learning complexity
  while enabling nuanced integration of crucial background context. We also
  introduce a novel target region ID resampling technique that enables any-length
  video inpainting, greatly enhancing our practical applicability. Additionally,
  we establish a scalable dataset pipeline leveraging current vision
  understanding models, contributing VPData and VPBench to facilitate
  segmentation-based inpainting training and assessment, the largest video
  inpainting dataset and benchmark to date with over 390K diverse clips. Using
  inpainting as a pipeline basis, we also explore downstream applications
  including video editing and video editing pair data generation, demonstrating
  competitive performance and significant practical potential. Extensive
  experiments demonstrate VideoPainter's superior performance in both any-length
  video inpainting and editing, across eight key metrics, including video
  quality, mask region preservation, and textual coherence.
paper_ai_keywords: dual-stream paradigm VideoPainter context encoder backbone parameters pre-trained video DiT semantic consistency target region ID resampling technique VPData VPBench segmentation-based inpainting video editing video editing pair data generation video quality mask region preservation textual coherence