Robust Shape Fitting for 3D Scene Abstraction
<|reference_start|>Robust Shape Fitting for 3D Scene Abstraction: Humans perceive and construct the world as an arrangement of simple parametric models. In particular, we can often describe man-made environments using volumetric primitives such as cuboids or cylinders. Inferring these primitives is important for attaining high-level, abstract scene descriptions. Previous approaches for primitive-based abstraction estimate shape parameters directly and are only able to reproduce simple objects. In contrast, we propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to a depth map. We condition the network on previously detected parts of the scene, parsing it one-by-one. To obtain cuboids from single RGB images, we additionally optimise a depth estimation CNN end-to-end. Naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene. We thus propose an improved occlusion-aware distance metric correctly handling opaque scenes. Furthermore, we present a neural network based cuboid solver which provides more parsimonious scene abstractions while also reducing inference time. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.<|reference_end|>
@article{kluger2024robust,
    title={Robust Shape Fitting for 3D Scene Abstraction},
    author={Florian Kluger, Eric Brachmann, Michael Ying Yang, Bodo Rosenhahn},
    journal={arXiv preprint arXiv:2403.10452},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.10452},
    primaryClass={cs.CV}
}