HaLo-NeRF: Learning Geometry-Guided Semantics for Exploring
  Unconstrained Photo Collections
Internet image collections containing photos captured by crowds of
photographers show promise for enabling digital exploration of large-scale
tourist landmarks. However, prior works focus primarily on geometric
reconstruction and visualization, neglecting the key role of language in
providing a semantic interface for navigation and fine-grained understanding.
In constrained 3D domains, recent methods have leveraged vision-and-language
models as a strong prior of 2D visual semantics. While these models display an
excellent understanding of broad visual semantics, they struggle with
unconstrained photo collections depicting such tourist landmarks, as they lack
expert knowledge of the architectural domain. In this work, we present a
localization system that connects neural representations of scenes depicting
large-scale landmarks with text describing a semantic region within the scene,
by harnessing the power of SOTA vision-and-language models with adaptations for
understanding landmark scene semantics. To bolster such models with
fine-grained knowledge, we leverage large-scale Internet data containing images
of similar landmarks along with weakly-related textual information. Our
approach is built upon the premise that images physically grounded in space can
provide a powerful supervision signal for localizing new concepts, whose
semantics may be unlocked from Internet textual metadata with large language
models. We use correspondences between views of scenes to bootstrap spatial
understanding of these semantics, providing guidance for 3D-compatible
segmentation that ultimately lifts to a volumetric scene representation. Our
results show that HaLo-NeRF can accurately localize a variety of semantic
concepts related to architectural landmarks, surpassing the results of other 3D
models as well as strong 2D segmentation baselines. Our project page is at
https://tau-vailab.github.io/HaLo-NeRF/.
662ef5bc5cbdc931c52ccd1a
Chen Dudai
admin_assigned
2024-04-29T12:11:25.706Z
6344298773b4f9cedab52048
/avatars/b239c57b6a281c7404104134b07e5423.svg
Chen Dudai
user
ChenDudai
662ef5bc5cbdc931c52ccd1b
Morris Alper
admin_assigned
2024-04-29T12:11:32.083Z
625d87e71decffbc2db8c314
/avatars/d0368008ec3c5438c96aba4c0ca05941.svg
Morris Alper
user
malper
662ef5bc5cbdc931c52ccd1c
Hana Bezalel
662ef5bc5cbdc931c52ccd1d
Rana Hanocka
admin_assigned
2024-04-29T12:11:41.401Z
630495ffce6b12280b193f30
/avatars/55b201131fa798c2a33d17dcab72fd1f.svg
Rana Hanocka
user
ranahanocka
662ef5bc5cbdc931c52ccd1e
Itai Lang
admin_assigned
2024-04-29T12:11:48.645Z
649972745e8b0d3f584ef26e
https://cdn-avatars.huggingface.co/v1/production/uploads/649972745e8b0d3f584ef26e/2k5DwtPm7fGCSIn5JWKUV.png
Itai Lang
user
itailang
662ef5bc5cbdc931c52ccd1f
Hadar Averbuch-Elor
HaLo-NeRF: Learning Geometry-Guided Semantics for Exploring
  Unconstrained Photo Collections
Internet image collections containing photos captured by crowds of
photographers show promise for enabling digital exploration of large-scale
tourist landmarks. However, prior works focus primarily on geometric
reconstruction and visualization, neglecting the key role of language in
providing a semantic interface for navigation and fine-grained understanding.
In constrained 3D domains, recent methods have leveraged vision-and-language
models as a strong prior of 2D visual semantics. While these models display an
excellent understanding of broad visual semantics, they struggle with
unconstrained photo collections depicting such tourist landmarks, as they lack
expert knowledge of the architectural domain. In this work, we present a
localization system that connects neural representations of scenes depicting
large-scale landmarks with text describing a semantic region within the scene,
by harnessing the power of SOTA vision-and-language models with adaptations for
understanding landmark scene semantics. To bolster such models with
fine-grained knowledge, we leverage large-scale Internet data containing images
of similar landmarks along with weakly-related textual information. Our
approach is built upon the premise that images physically grounded in space can
provide a powerful supervision signal for localizing new concepts, whose
semantics may be unlocked from Internet textual metadata with large language
models. We use correspondences between views of scenes to bootstrap spatial
understanding of these semantics, providing guidance for 3D-compatible
segmentation that ultimately lifts to a volumetric scene representation. Our
results show that HaLo-NeRF can accurately localize a variety of semantic
concepts related to architectural landmarks, surpassing the results of other 3D
models as well as strong 2D segmentation baselines. Our project page is at
https://tau-vailab.github.io/HaLo-NeRF/.
A localization system using vision-and-language models enhances understanding of architectural landmarks by integrating textual and image data, improving localization accuracy over other 3D and 2D models.
vision-and-language models local representations HaLo-NeRF spatial understanding volumetric scene representation semantic localization 3D-compatible segmentation weakly-related textual information geometric reconstruction semantic interface