Zero-Shot Multi-Object Scene Completion
<|reference_start|>Zero-Shot Multi-Object Scene Completion: We present a 3D scene completion method that recovers the complete geometry of multiple unseen objects in complex scenes from a single RGB-D image. Despite notable advancements in single-object 3D shape completion, high-quality reconstructions in highly cluttered real-world multi-object scenes remains a challenge. To address this issue, we propose OctMAE, an architecture that leverages an Octree U-Net and a latent 3D MAE to achieve high-quality and near real-time multi-object scene completion through both local and global geometric reasoning. Because a naive 3D MAE can be computationally intractable and memory intensive even in the latent space, we introduce a novel occlusion masking strategy and adopt 3D rotary embeddings, which significantly improves the runtime and scene completion quality. To generalize to a wide range of objects in diverse scenes, we create a large-scale photorealistic dataset, featuring a diverse set of 12K 3D object models from the Objaverse dataset which are rendered in multi-object scenes with physics-based positioning. Our method outperforms the current state-of-the-art on both synthetic and real-world datasets and demonstrates a strong zero-shot capability.<|reference_end|>
@article{iwase2024zero-shot,
    title={Zero-Shot Multi-Object Scene Completion},
    author={Shun Iwase, Katherine Liu, Vitor Guizilini, Adrien Gaidon, Kris
  Kitani, Rares Ambrus, and Sergey Zakharov},
    journal={arXiv preprint arXiv:2403.14628},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.14628},
    primaryClass={cs.CV}
}