title: OpenMask3D: Open-Vocabulary 3D Instance Segmentation
summary: We introduce the task of open-vocabulary 3D instance segmentation.
  Traditional approaches for 3D instance segmentation largely rely on existing 3D
  annotated datasets, which are restricted to a closed-set of object categories.
  This is an important limitation for real-life applications where one might need
  to perform tasks guided by novel, open-vocabulary queries related to objects
  from a wide variety. Recently, open-vocabulary 3D scene understanding methods
  have emerged to address this problem by learning queryable features per each
  point in the scene. While such a representation can be directly employed to
  perform semantic segmentation, existing methods have limitations in their
  ability to identify object instances. In this work, we address this limitation,
  and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D
  instance segmentation. Guided by predicted class-agnostic 3D instance masks,
  our model aggregates per-mask features via multi-view fusion of CLIP-based
  image embeddings. We conduct experiments and ablation studies on the ScanNet200
  dataset to evaluate the performance of OpenMask3D, and provide insights about
  the open-vocabulary 3D instance segmentation task. We show that our approach
  outperforms other open-vocabulary counterparts, particularly on the long-tail
  distribution. Furthermore, OpenMask3D goes beyond the limitations of
  close-vocabulary approaches, and enables the segmentation of object instances
  based on free-form queries describing object properties such as semantics,
  geometry, affordances, and material properties.
paper_authors._id: 64991eef7c15c218be927128
paper_authors.name: Ayça Takmaz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-27T07:27:47.817Z
paper_authors._id: 6499631f906b89ee66f1fb81
paper_authors.avatarUrl: /avatars/fe27062dfc20cf57169d596457d0ef92.svg
paper_authors.fullname: Ayca Takmaz
paper_authors.type: user
paper_authors.user: aycatakmaz
paper_authors._id: 64991eef7c15c218be927129
paper_authors.name: Elisabetta Fedele
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-27T07:27:52.704Z
paper_authors._id: 62df0ae635a8bfe88fc4ac80
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62df0ae635a8bfe88fc4ac80/yWu5Kfq_88Ni2uy3HvnLe.jpeg
paper_authors.fullname: elisabetta fedele
paper_authors.type: user
paper_authors.user: efedele
paper_authors._id: 64991eef7c15c218be92712a
paper_authors.name: Robert W. Sumner
paper_authors._id: 64991eef7c15c218be92712b
paper_authors.name: Marc Pollefeys
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:43:25.121Z
paper_authors._id: 67b5fa179782a5e2fd2cb26a
paper_authors.avatarUrl: /avatars/62c38f29ec641e001eeddf840bea21a0.svg
paper_authors.fullname: Marc Pollefeys
paper_authors.type: user
paper_authors.user: mapo1
paper_authors._id: 64991eef7c15c218be92712c
paper_authors.name: Federico Tombari
paper_authors._id: 64991eef7c15c218be92712d
paper_authors.name: Francis Engelmann
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-18T08:48:44.180Z
paper_authors._id: 671a7160e45d92e304e82f45
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-trHIg97qX0dE9T4UYycc.png
paper_authors.fullname: Francis Engelmann
paper_authors.type: user
paper_authors.user: francisengelmann
paper_title: OpenMask3D: Open-Vocabulary 3D Instance Segmentation
paper_summary: We introduce the task of open-vocabulary 3D instance segmentation.
  Traditional approaches for 3D instance segmentation largely rely on existing 3D
  annotated datasets, which are restricted to a closed-set of object categories.
  This is an important limitation for real-life applications where one might need
  to perform tasks guided by novel, open-vocabulary queries related to objects
  from a wide variety. Recently, open-vocabulary 3D scene understanding methods
  have emerged to address this problem by learning queryable features per each
  point in the scene. While such a representation can be directly employed to
  perform semantic segmentation, existing methods have limitations in their
  ability to identify object instances. In this work, we address this limitation,
  and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D
  instance segmentation. Guided by predicted class-agnostic 3D instance masks,
  our model aggregates per-mask features via multi-view fusion of CLIP-based
  image embeddings. We conduct experiments and ablation studies on the ScanNet200
  dataset to evaluate the performance of OpenMask3D, and provide insights about
  the open-vocabulary 3D instance segmentation task. We show that our approach
  outperforms other open-vocabulary counterparts, particularly on the long-tail
  distribution. Furthermore, OpenMask3D goes beyond the limitations of
  close-vocabulary approaches, and enables the segmentation of object instances
  based on free-form queries describing object properties such as semantics,
  geometry, affordances, and material properties.
paper_ai_summary: OpenMask3D is a zero-shot approach for open-vocabulary 3D instance segmentation using class-agnostic instance masks and multi-view fusion of CLIP-based image embeddings.
paper_ai_keywords: open-vocabulary 3D instance segmentation 3D instance masks multi-view fusion CLIP-based image embeddings