QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding
<|reference_start|>QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding: Understanding the structural organisation of 3D indoor scenes in terms of rooms is often accomplished via floorplan extraction. Robotic tasks such as planning and navigation require a semantic understanding of the scene as well. This is typically achieved via object-level semantic segmentation. However, such methods struggle to segment out topological regions like "kitchen" in the scene. In this work, we introduce a two-step pipeline. First, we extract a topological map, i.e., floorplan of the indoor scene using a novel multi-channel occupancy representation. Then, we generate CLIP-aligned features and semantic labels for every room instance based on the objects it contains using a self-attention transformer. Our language-topology alignment supports natural language querying, e.g., a "place to cook" locates the "kitchen". We outperform the current state-of-the-art on room segmentation by ~20% and room classification by ~12%. Our detailed qualitative analysis and ablation studies provide insights into the problem of joint structural and semantic 3D scene understanding.<|reference_end|>
@article{mehan2024questmaps:,
    title={QueSTMaps: Queryable Semantic Topological Maps for 3D Scene
  Understanding},
    author={Yash Mehan, Kumaraditya Gupta, Rohit Jayanti, Anirudh Govil, Sourav
  Garg, Madhava Krishna},
    journal={arXiv preprint arXiv:2404.06442},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.06442},
    primaryClass={cs.CV cs.RO}
}