title: HaLo-NeRF: Learning Geometry-Guided Semantics for Exploring
    Unconstrained Photo Collections
summary: Internet image collections containing photos captured by crowds of
  photographers show promise for enabling digital exploration of large-scale
  tourist landmarks. However, prior works focus primarily on geometric
  reconstruction and visualization, neglecting the key role of language in
  providing a semantic interface for navigation and fine-grained understanding.
  In constrained 3D domains, recent methods have leveraged vision-and-language
  models as a strong prior of 2D visual semantics. While these models display an
  excellent understanding of broad visual semantics, they struggle with
  unconstrained photo collections depicting such tourist landmarks, as they lack
  expert knowledge of the architectural domain. In this work, we present a
  localization system that connects neural representations of scenes depicting
  large-scale landmarks with text describing a semantic region within the scene,
  by harnessing the power of SOTA vision-and-language models with adaptations for
  understanding landmark scene semantics. To bolster such models with
  fine-grained knowledge, we leverage large-scale Internet data containing images
  of similar landmarks along with weakly-related textual information. Our
  approach is built upon the premise that images physically grounded in space can
  provide a powerful supervision signal for localizing new concepts, whose
  semantics may be unlocked from Internet textual metadata with large language
  models. We use correspondences between views of scenes to bootstrap spatial
  understanding of these semantics, providing guidance for 3D-compatible
  segmentation that ultimately lifts to a volumetric scene representation. Our
  results show that HaLo-NeRF can accurately localize a variety of semantic
  concepts related to architectural landmarks, surpassing the results of other 3D
  models as well as strong 2D segmentation baselines. Our project page is at
  https://tau-vailab.github.io/HaLo-NeRF/.
paper_authors._id: 662ef5bc5cbdc931c52ccd1a
paper_authors.name: Chen Dudai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T12:11:25.706Z
paper_authors._id: 6344298773b4f9cedab52048
paper_authors.avatarUrl: /avatars/b239c57b6a281c7404104134b07e5423.svg
paper_authors.fullname: Chen Dudai
paper_authors.type: user
paper_authors.user: ChenDudai
paper_authors._id: 662ef5bc5cbdc931c52ccd1b
paper_authors.name: Morris Alper
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T12:11:32.083Z
paper_authors._id: 625d87e71decffbc2db8c314
paper_authors.avatarUrl: /avatars/d0368008ec3c5438c96aba4c0ca05941.svg
paper_authors.fullname: Morris Alper
paper_authors.type: user
paper_authors.user: malper
paper_authors._id: 662ef5bc5cbdc931c52ccd1c
paper_authors.name: Hana Bezalel
paper_authors._id: 662ef5bc5cbdc931c52ccd1d
paper_authors.name: Rana Hanocka
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T12:11:41.401Z
paper_authors._id: 630495ffce6b12280b193f30
paper_authors.avatarUrl: /avatars/55b201131fa798c2a33d17dcab72fd1f.svg
paper_authors.fullname: Rana Hanocka
paper_authors.type: user
paper_authors.user: ranahanocka
paper_authors._id: 662ef5bc5cbdc931c52ccd1e
paper_authors.name: Itai Lang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T12:11:48.645Z
paper_authors._id: 649972745e8b0d3f584ef26e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649972745e8b0d3f584ef26e/2k5DwtPm7fGCSIn5JWKUV.png
paper_authors.fullname: Itai Lang
paper_authors.type: user
paper_authors.user: itailang
paper_authors._id: 662ef5bc5cbdc931c52ccd1f
paper_authors.name: Hadar Averbuch-Elor
paper_title: HaLo-NeRF: Learning Geometry-Guided Semantics for Exploring
    Unconstrained Photo Collections
paper_summary: Internet image collections containing photos captured by crowds of
  photographers show promise for enabling digital exploration of large-scale
  tourist landmarks. However, prior works focus primarily on geometric
  reconstruction and visualization, neglecting the key role of language in
  providing a semantic interface for navigation and fine-grained understanding.
  In constrained 3D domains, recent methods have leveraged vision-and-language
  models as a strong prior of 2D visual semantics. While these models display an
  excellent understanding of broad visual semantics, they struggle with
  unconstrained photo collections depicting such tourist landmarks, as they lack
  expert knowledge of the architectural domain. In this work, we present a
  localization system that connects neural representations of scenes depicting
  large-scale landmarks with text describing a semantic region within the scene,
  by harnessing the power of SOTA vision-and-language models with adaptations for
  understanding landmark scene semantics. To bolster such models with
  fine-grained knowledge, we leverage large-scale Internet data containing images
  of similar landmarks along with weakly-related textual information. Our
  approach is built upon the premise that images physically grounded in space can
  provide a powerful supervision signal for localizing new concepts, whose
  semantics may be unlocked from Internet textual metadata with large language
  models. We use correspondences between views of scenes to bootstrap spatial
  understanding of these semantics, providing guidance for 3D-compatible
  segmentation that ultimately lifts to a volumetric scene representation. Our
  results show that HaLo-NeRF can accurately localize a variety of semantic
  concepts related to architectural landmarks, surpassing the results of other 3D
  models as well as strong 2D segmentation baselines. Our project page is at
  https://tau-vailab.github.io/HaLo-NeRF/.
paper_ai_keywords: NeRF vision-and-language models semantic interface spatial understanding segmentation volumetric scene representation