title: ProPainter: Improving Propagation and Transformer for Video Inpainting
summary: Flow-based propagation and spatiotemporal Transformer are two mainstream
  mechanisms in video inpainting (VI). Despite the effectiveness of these
  components, they still suffer from some limitations that affect their
  performance. Previous propagation-based approaches are performed separately
  either in the image or feature domain. Global image propagation isolated from
  learning may cause spatial misalignment due to inaccurate optical flow.
  Moreover, memory or computational constraints limit the temporal range of
  feature propagation and video Transformer, preventing exploration of
  correspondence information from distant frames. To address these issues, we
  propose an improved framework, called ProPainter, which involves enhanced
  ProPagation and an efficient Transformer. Specifically, we introduce
  dual-domain propagation that combines the advantages of image and feature
  warping, exploiting global correspondences reliably. We also propose a
  mask-guided sparse video Transformer, which achieves high efficiency by
  discarding unnecessary and redundant tokens. With these components, ProPainter
  outperforms prior arts by a large margin of 1.46 dB in PSNR while maintaining
  appealing efficiency.
paper_authors._id: 64fab5f914636d417a8c0d11
paper_authors.name: Shangchen Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:45:32.479Z
paper_authors._id: 62e57662ae9d3f10acbb1b1b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e57662ae9d3f10acbb1b1b/lg58jdbNyv6LGH2LFnZDF.png
paper_authors.fullname: Shangchen Zhou
paper_authors.type: user
paper_authors.user: sczhou
paper_authors._id: 64fab5f914636d417a8c0d12
paper_authors.name: Chongyi Li
paper_authors._id: 64fab5f914636d417a8c0d13
paper_authors.name: Kelvin C. K. Chan
paper_authors._id: 64fab5f914636d417a8c0d14
paper_authors.name: Chen Change Loy
paper_title: ProPainter: Improving Propagation and Transformer for Video Inpainting
paper_summary: Flow-based propagation and spatiotemporal Transformer are two mainstream
  mechanisms in video inpainting (VI). Despite the effectiveness of these
  components, they still suffer from some limitations that affect their
  performance. Previous propagation-based approaches are performed separately
  either in the image or feature domain. Global image propagation isolated from
  learning may cause spatial misalignment due to inaccurate optical flow.
  Moreover, memory or computational constraints limit the temporal range of
  feature propagation and video Transformer, preventing exploration of
  correspondence information from distant frames. To address these issues, we
  propose an improved framework, called ProPainter, which involves enhanced
  ProPagation and an efficient Transformer. Specifically, we introduce
  dual-domain propagation that combines the advantages of image and feature
  warping, exploiting global correspondences reliably. We also propose a
  mask-guided sparse video Transformer, which achieves high efficiency by
  discarding unnecessary and redundant tokens. With these components, ProPainter
  outperforms prior arts by a large margin of 1.46 dB in PSNR while maintaining
  appealing efficiency.
paper_ai_summary: ProPainter is a framework that improves video inpainting by combining dual-domain propagation and a mask-guided sparse video Transformer, outperforming existing methods in PSNR while maintaining efficiency.
paper_ai_keywords: dual-domain propagation image warping feature warping global correspondences mask-guided sparse video Transformer