Masked Scene Modeling: Narrowing the Gap Between Supervised and
  Self-Supervised Learning in 3D Scene Understanding
Self-supervised learning has transformed 2D computer vision by enabling
models trained on large, unannotated datasets to provide versatile
off-the-shelf features that perform similarly to models trained with labels.
However, in 3D scene understanding, self-supervised methods are typically only
used as a weight initialization step for task-specific fine-tuning, limiting
their utility for general-purpose feature extraction. This paper addresses this
shortcoming by proposing a robust evaluation protocol specifically designed to
assess the quality of self-supervised features for 3D scene understanding. Our
protocol uses multi-resolution feature sampling of hierarchical models to
create rich point-level representations that capture the semantic capabilities
of the model and, hence, are suitable for evaluation with linear probing and
nearest-neighbor methods. Furthermore, we introduce the first self-supervised
model that performs similarly to supervised models when only off-the-shelf
features are used in a linear probing setup. In particular, our model is
trained natively in 3D with a novel self-supervised approach based on a Masked
Scene Modeling objective, which reconstructs deep features of masked patches in
a bottom-up manner and is specifically tailored to hierarchical 3D models. Our
experiments not only demonstrate that our method achieves competitive
performance to supervised models, but also surpasses existing self-supervised
approaches by a large margin. The model and training code can be found at our
Github repository (https://github.com/phermosilla/msm).
67f7932d7ce01a821788dadb
Pedro Hermosilla
claimed_verified
2025-04-10T09:57:52.157Z
67a37342e7acfe540dcd5f0e
/avatars/759a4769ab390072e62859f5c1db3c0a.svg
Pedro Hermosilla
user
phermosilla
67f7932d7ce01a821788dadc
Christian Stippel
67f7932d7ce01a821788dadd
Leon Sick
claimed_verified
2025-04-10T13:23:18.469Z
642150c201c62c1e41fc2390
https://cdn-avatars.huggingface.co/v1/production/uploads/642150c201c62c1e41fc2390/4udBai02RARpIujIeiCX-.png
Leon Sick
user
leonsick
Masked Scene Modeling: Narrowing the Gap Between Supervised and
  Self-Supervised Learning in 3D Scene Understanding
Self-supervised learning has transformed 2D computer vision by enabling
models trained on large, unannotated datasets to provide versatile
off-the-shelf features that perform similarly to models trained with labels.
However, in 3D scene understanding, self-supervised methods are typically only
used as a weight initialization step for task-specific fine-tuning, limiting
their utility for general-purpose feature extraction. This paper addresses this
shortcoming by proposing a robust evaluation protocol specifically designed to
assess the quality of self-supervised features for 3D scene understanding. Our
protocol uses multi-resolution feature sampling of hierarchical models to
create rich point-level representations that capture the semantic capabilities
of the model and, hence, are suitable for evaluation with linear probing and
nearest-neighbor methods. Furthermore, we introduce the first self-supervised
model that performs similarly to supervised models when only off-the-shelf
features are used in a linear probing setup. In particular, our model is
trained natively in 3D with a novel self-supervised approach based on a Masked
Scene Modeling objective, which reconstructs deep features of masked patches in
a bottom-up manner and is specifically tailored to hierarchical 3D models. Our
experiments not only demonstrate that our method achieves competitive
performance to supervised models, but also surpasses existing self-supervised
approaches by a large margin. The model and training code can be found at our
Github repository (https://github.com/phermosilla/msm).
A self-supervised method for 3D scene understanding using masked scene modeling achieves competitive performance with linear probing, surpassing existing self-supervised approaches.
self-supervised learning 3D scene understanding weight initialization task-specific fine-tuning feature extraction evaluation protocol multi-resolution feature sampling hierarchical models point-level representations semantic capabilities linear probing nearest-neighbor methods Masked Scene Modeling hierarchical 3D models bottom-up manner deep features masked patches