View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields
<|reference_start|>View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields: Large-scale vision foundation models such as Segment Anything (SAM) demonstrate impressive performance in zero-shot image segmentation at multiple levels of granularity. However, these zero-shot predictions are rarely 3D-consistent. As the camera viewpoint changes in a scene, so do the segmentation predictions, as well as the characterizations of "coarse" or "fine" granularity. In this work, we address the challenging task of lifting multi-granular and view-inconsistent image segmentations into a hierarchical and 3D-consistent representation. We learn a novel feature field within a Neural Radiance Field (NeRF) representing a 3D scene, whose segmentation structure can be revealed at different scales by simply using different thresholds on feature distance. Our key idea is to learn an ultrametric feature space, which unlike a Euclidean space, exhibits transitivity in distance-based grouping, naturally leading to a hierarchical clustering. Put together, our method takes view-inconsistent multi-granularity 2D segmentations as input and produces a hierarchy of 3D-consistent segmentations as output. We evaluate our method and several baselines on synthetic datasets with multi-view images and multi-granular segmentation, showcasing improved accuracy and viewpoint-consistency. We additionally provide qualitative examples of our model's 3D hierarchical segmentations in real world scenes. The code and dataset are available at https://github.com/hardyho/ultrametric_feature_fields<|reference_end|>
@article{he2024view-consistent,
    title={View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature
  Fields},
    author={Haodi He, Colton Stearns, Adam W. Harley, Leonidas J. Guibas},
    journal={arXiv preprint arXiv:2405.19678},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.19678},
    primaryClass={cs.CV cs.AI}
}