title: Masked Scene Modeling: Narrowing the Gap Between Supervised and
    Self-Supervised Learning in 3D Scene Understanding
summary: Self-supervised learning has transformed 2D computer vision by enabling
  models trained on large, unannotated datasets to provide versatile
  off-the-shelf features that perform similarly to models trained with labels.
  However, in 3D scene understanding, self-supervised methods are typically only
  used as a weight initialization step for task-specific fine-tuning, limiting
  their utility for general-purpose feature extraction. This paper addresses this
  shortcoming by proposing a robust evaluation protocol specifically designed to
  assess the quality of self-supervised features for 3D scene understanding. Our
  protocol uses multi-resolution feature sampling of hierarchical models to
  create rich point-level representations that capture the semantic capabilities
  of the model and, hence, are suitable for evaluation with linear probing and
  nearest-neighbor methods. Furthermore, we introduce the first self-supervised
  model that performs similarly to supervised models when only off-the-shelf
  features are used in a linear probing setup. In particular, our model is
  trained natively in 3D with a novel self-supervised approach based on a Masked
  Scene Modeling objective, which reconstructs deep features of masked patches in
  a bottom-up manner and is specifically tailored to hierarchical 3D models. Our
  experiments not only demonstrate that our method achieves competitive
  performance to supervised models, but also surpasses existing self-supervised
  approaches by a large margin. The model and training code can be found at our
  Github repository (https://github.com/phermosilla/msm).
paper_authors._id: 67f7932d7ce01a821788dadb
paper_authors.name: Pedro Hermosilla
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T09:57:52.157Z
paper_authors._id: 67a37342e7acfe540dcd5f0e
paper_authors.avatarUrl: /avatars/759a4769ab390072e62859f5c1db3c0a.svg
paper_authors.fullname: Pedro Hermosilla
paper_authors.type: user
paper_authors.user: phermosilla
paper_authors._id: 67f7932d7ce01a821788dadc
paper_authors.name: Christian Stippel
paper_authors._id: 67f7932d7ce01a821788dadd
paper_authors.name: Leon Sick
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T13:23:18.469Z
paper_authors._id: 642150c201c62c1e41fc2390
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642150c201c62c1e41fc2390/4udBai02RARpIujIeiCX-.png
paper_authors.fullname: Leon Sick
paper_authors.type: user
paper_authors.user: leonsick
paper_title: Masked Scene Modeling: Narrowing the Gap Between Supervised and
    Self-Supervised Learning in 3D Scene Understanding
paper_summary: Self-supervised learning has transformed 2D computer vision by enabling
  models trained on large, unannotated datasets to provide versatile
  off-the-shelf features that perform similarly to models trained with labels.
  However, in 3D scene understanding, self-supervised methods are typically only
  used as a weight initialization step for task-specific fine-tuning, limiting
  their utility for general-purpose feature extraction. This paper addresses this
  shortcoming by proposing a robust evaluation protocol specifically designed to
  assess the quality of self-supervised features for 3D scene understanding. Our
  protocol uses multi-resolution feature sampling of hierarchical models to
  create rich point-level representations that capture the semantic capabilities
  of the model and, hence, are suitable for evaluation with linear probing and
  nearest-neighbor methods. Furthermore, we introduce the first self-supervised
  model that performs similarly to supervised models when only off-the-shelf
  features are used in a linear probing setup. In particular, our model is
  trained natively in 3D with a novel self-supervised approach based on a Masked
  Scene Modeling objective, which reconstructs deep features of masked patches in
  a bottom-up manner and is specifically tailored to hierarchical 3D models. Our
  experiments not only demonstrate that our method achieves competitive
  performance to supervised models, but also surpasses existing self-supervised
  approaches by a large margin. The model and training code can be found at our
  Github repository (https://github.com/phermosilla/msm).
paper_ai_summary: A self-supervised method for 3D scene understanding using masked scene modeling achieves competitive performance with linear probing, surpassing existing self-supervised approaches.
paper_ai_keywords: self-supervised learning 3D scene understanding weight initialization task-specific fine-tuning feature extraction evaluation protocol multi-resolution feature sampling hierarchical models point-level representations semantic capabilities linear probing nearest-neighbor methods Masked Scene Modeling hierarchical 3D models bottom-up manner deep features masked patches