title: MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and
    3D Editing
summary: Novel View Synthesis (NVS) and 3D generation have recently achieved prominent
  improvements. However, these works mainly focus on confined categories or
  synthetic 3D assets, which are discouraged from generalizing to challenging
  in-the-wild scenes and fail to be employed with 2D synthesis directly.
  Moreover, these methods heavily depended on camera poses, limiting their
  real-world applications. To overcome these issues, we propose MVInpainter,
  re-formulating the 3D editing as a multi-view 2D inpainting task. Specifically,
  MVInpainter partially inpaints multi-view images with the reference guidance
  rather than intractably generating an entirely novel view from scratch, which
  largely simplifies the difficulty of in-the-wild NVS and leverages unmasked
  clues instead of explicit pose conditions. To ensure cross-view consistency,
  MVInpainter is enhanced by video priors from motion components and appearance
  guidance from concatenated reference key&value attention. Furthermore,
  MVInpainter incorporates slot attention to aggregate high-level optical flow
  features from unmasked regions to control the camera movement with pose-free
  training and inference. Sufficient scene-level experiments on both
  object-centric and forward-facing datasets verify the effectiveness of
  MVInpainter, including diverse tasks, such as multi-view object removal,
  synthesis, insertion, and replacement. The project page is
  https://ewrfcas.github.io/MVInpainter/.
paper_authors._id: 66bec0f7ae70890c90f59f84
paper_authors.name: Chenjie Cao
paper_authors._id: 66bec0f7ae70890c90f59f85
paper_authors.name: Chaohui Yu
paper_authors._id: 66bec0f7ae70890c90f59f86
paper_authors.name: Yanwei Fu
paper_authors._id: 66bec0f7ae70890c90f59f87
paper_authors.name: Fan Wang
paper_authors._id: 66bec0f7ae70890c90f59f88
paper_authors.name: Xiangyang Xue
paper_title: MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and
    3D Editing
paper_summary: Novel View Synthesis (NVS) and 3D generation have recently achieved prominent
  improvements. However, these works mainly focus on confined categories or
  synthetic 3D assets, which are discouraged from generalizing to challenging
  in-the-wild scenes and fail to be employed with 2D synthesis directly.
  Moreover, these methods heavily depended on camera poses, limiting their
  real-world applications. To overcome these issues, we propose MVInpainter,
  re-formulating the 3D editing as a multi-view 2D inpainting task. Specifically,
  MVInpainter partially inpaints multi-view images with the reference guidance
  rather than intractably generating an entirely novel view from scratch, which
  largely simplifies the difficulty of in-the-wild NVS and leverages unmasked
  clues instead of explicit pose conditions. To ensure cross-view consistency,
  MVInpainter is enhanced by video priors from motion components and appearance
  guidance from concatenated reference key&value attention. Furthermore,
  MVInpainter incorporates slot attention to aggregate high-level optical flow
  features from unmasked regions to control the camera movement with pose-free
  training and inference. Sufficient scene-level experiments on both
  object-centric and forward-facing datasets verify the effectiveness of
  MVInpainter, including diverse tasks, such as multi-view object removal,
  synthesis, insertion, and replacement. The project page is
  https://ewrfcas.github.io/MVInpainter/.
paper_ai_keywords: View Synthesis 3D generation multi-view inpainting camera poses reference guidance video priors motion components appearance guidance key&value attention slot attention high-level optical flow features scene-level experiments object-centric datasets forward-facing datasets multi-view object removal synthesis insertion replacement