NeRF-Guided Unsupervised Learning of RGB-D Registration
<|reference_start|>NeRF-Guided Unsupervised Learning of RGB-D Registration: This paper focuses on training a robust RGB-D registration model without ground-truth pose supervision. Existing methods usually adopt a pairwise training strategy based on differentiable rendering, which enforces the photometric and the geometric consistency between the two registered frames as supervision. However, this frame-to-frame framework suffers from poor multi-view consistency due to factors such as lighting changes, geometry occlusion and reflective materials. In this paper, we present NeRF-UR, a novel frame-to-model optimization framework for unsupervised RGB-D registration. Instead of frame-to-frame consistency, we leverage the neural radiance field (NeRF) as a global model of the scene and use the consistency between the input and the NeRF-rerendered frames for pose optimization. This design can significantly improve the robustness in scenarios with poor multi-view consistency and provides better learning signal for the registration model. Furthermore, to bootstrap the NeRF optimization, we create a synthetic dataset, Sim-RGBD, through a photo-realistic simulator to warm up the registration model. By first training the registration model on Sim-RGBD and later unsupervisedly fine-tuning on real data, our framework enables distilling the capability of feature extraction and registration from simulation to reality. Our method outperforms the state-of-the-art counterparts on two popular indoor RGB-D datasets, ScanNet and 3DMatch. Code and models will be released for paper reproduction.<|reference_end|>
@article{yu2024nerf-guided,
    title={NeRF-Guided Unsupervised Learning of RGB-D Registration},
    author={Zhinan Yu and Zheng Qin and Yijie Tang and Yongjun Wang and Renjiao Yi
  and Chenyang Zhu and Kai Xu},
    journal={arXiv preprint arXiv:2405.00507},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.00507},
    primaryClass={cs.CV}
}