OpenMask3D: Open-Vocabulary 3D Instance Segmentation
We introduce the task of open-vocabulary 3D instance segmentation.
Traditional approaches for 3D instance segmentation largely rely on existing 3D
annotated datasets, which are restricted to a closed-set of object categories.
This is an important limitation for real-life applications where one might need
to perform tasks guided by novel, open-vocabulary queries related to objects
from a wide variety. Recently, open-vocabulary 3D scene understanding methods
have emerged to address this problem by learning queryable features per each
point in the scene. While such a representation can be directly employed to
perform semantic segmentation, existing methods have limitations in their
ability to identify object instances. In this work, we address this limitation,
and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D
instance segmentation. Guided by predicted class-agnostic 3D instance masks,
our model aggregates per-mask features via multi-view fusion of CLIP-based
image embeddings. We conduct experiments and ablation studies on the ScanNet200
dataset to evaluate the performance of OpenMask3D, and provide insights about
the open-vocabulary 3D instance segmentation task. We show that our approach
outperforms other open-vocabulary counterparts, particularly on the long-tail
distribution. Furthermore, OpenMask3D goes beyond the limitations of
close-vocabulary approaches, and enables the segmentation of object instances
based on free-form queries describing object properties such as semantics,
geometry, affordances, and material properties.
64991eef7c15c218be927128
Ayça Takmaz
claimed_verified
2023-06-27T07:27:47.817Z
6499631f906b89ee66f1fb81
/avatars/fe27062dfc20cf57169d596457d0ef92.svg
Ayca Takmaz
user
aycatakmaz
64991eef7c15c218be927129
Elisabetta Fedele
claimed_verified
2023-06-27T07:27:52.704Z
62df0ae635a8bfe88fc4ac80
https://cdn-avatars.huggingface.co/v1/production/uploads/62df0ae635a8bfe88fc4ac80/yWu5Kfq_88Ni2uy3HvnLe.jpeg
elisabetta fedele
user
efedele
64991eef7c15c218be92712a
Robert W. Sumner
64991eef7c15c218be92712b
Marc Pollefeys
claimed_verified
2025-02-25T09:43:25.121Z
67b5fa179782a5e2fd2cb26a
/avatars/62c38f29ec641e001eeddf840bea21a0.svg
Marc Pollefeys
user
mapo1
64991eef7c15c218be92712c
Federico Tombari
64991eef7c15c218be92712d
Francis Engelmann
claimed_verified
2025-05-18T08:48:44.180Z
671a7160e45d92e304e82f45
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-trHIg97qX0dE9T4UYycc.png
Francis Engelmann
user
francisengelmann
OpenMask3D: Open-Vocabulary 3D Instance Segmentation
We introduce the task of open-vocabulary 3D instance segmentation.
Traditional approaches for 3D instance segmentation largely rely on existing 3D
annotated datasets, which are restricted to a closed-set of object categories.
This is an important limitation for real-life applications where one might need
to perform tasks guided by novel, open-vocabulary queries related to objects
from a wide variety. Recently, open-vocabulary 3D scene understanding methods
have emerged to address this problem by learning queryable features per each
point in the scene. While such a representation can be directly employed to
perform semantic segmentation, existing methods have limitations in their
ability to identify object instances. In this work, we address this limitation,
and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D
instance segmentation. Guided by predicted class-agnostic 3D instance masks,
our model aggregates per-mask features via multi-view fusion of CLIP-based
image embeddings. We conduct experiments and ablation studies on the ScanNet200
dataset to evaluate the performance of OpenMask3D, and provide insights about
the open-vocabulary 3D instance segmentation task. We show that our approach
outperforms other open-vocabulary counterparts, particularly on the long-tail
distribution. Furthermore, OpenMask3D goes beyond the limitations of
close-vocabulary approaches, and enables the segmentation of object instances
based on free-form queries describing object properties such as semantics,
geometry, affordances, and material properties.
OpenMask3D is a zero-shot approach for open-vocabulary 3D instance segmentation using class-agnostic instance masks and multi-view fusion of CLIP-based image embeddings.
open-vocabulary 3D instance segmentation 3D instance masks multi-view fusion CLIP-based image embeddings