PhotoReg: Photometrically Registering 3D Gaussian Splatting Models
<|reference_start|>PhotoReg: Photometrically Registering 3D Gaussian Splatting Models: Building accurate representations of the environment is critical for intelligent robots to make decisions during deployment. Advances in photorealistic environment models have enabled robots to develop hyper-realistic reconstructions, which can be used to generate images that are intuitive for human inspection. In particular, the recently introduced \ac{3DGS}, which describes the scene with up to millions of primitive ellipsoids, can be rendered in real time. \ac{3DGS} has rapidly gained prominence. However, a critical unsolved problem persists: how can we fuse multiple \ac{3DGS} into a single coherent model? Solving this problem will enable robot teams to jointly build \ac{3DGS} models of their surroundings. A key insight of this work is to leverage the {duality} between photorealistic reconstructions, which render realistic 2D images from 3D structure, and \emph{3D foundation models}, which predict 3D structure from image pairs. To this end, we develop PhotoReg, a framework to register multiple photorealistic \ac{3DGS} models with 3D foundation models. As \ac{3DGS} models are generally built from monocular camera images, they have \emph{arbitrary scale}. To resolve this, PhotoReg actively enforces scale consistency among the different \ac{3DGS} models by considering depth estimates within these models. Then, the alignment is iteratively refined with fine-grained photometric losses to produce high-quality fused \ac{3DGS} models. We rigorously evaluate PhotoReg on both standard benchmark datasets and our custom-collected datasets, including with two quadruped robots. The code is released at \url{ziweny11.github.io/photoreg}.<|reference_end|>
@article{yuan2024photoreg:,
    title={PhotoReg: Photometrically Registering 3D Gaussian Splatting Models},
    author={Ziwen Yuan, Tianyi Zhang, Matthew Johnson-Roberson, Weiming Zhi},
    journal={arXiv preprint arXiv:2410.05044},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.05044},
    primaryClass={cs.RO cs.AI cs.CV cs.LG}
}