Sonata: Self-Supervised Learning of Reliable Point Representations
In this paper, we question whether we have a reliable self-supervised point
cloud model that can be used for diverse 3D tasks via simple linear probing,
even with limited data and minimal computation. We find that existing 3D
self-supervised learning approaches fall short when evaluated on representation
quality through linear probing. We hypothesize that this is due to what we term
the "geometric shortcut", which causes representations to collapse to low-level
spatial features. This challenge is unique to 3D and arises from the sparse
nature of point cloud data. We address it through two key strategies: obscuring
spatial information and enhancing the reliance on input features, ultimately
composing a Sonata of 140k point clouds through self-distillation. Sonata is
simple and intuitive, yet its learned representations are strong and reliable:
zero-shot visualizations demonstrate semantic grouping, alongside strong
spatial reasoning through nearest-neighbor relationships. Sonata demonstrates
exceptional parameter and data efficiency, tripling linear probing accuracy
(from 21.8% to 72.5%) on ScanNet and nearly doubling performance with only 1%
of the data compared to previous approaches. Full fine-tuning further advances
SOTA across both 3D indoor and outdoor perception tasks.
67dd5935b4cce80ee64cd98f
Xiaoyang Wu
claimed_verified
2025-03-21T12:46:08.995Z
643e5d6a1d0e956d94bb3608
https://cdn-avatars.huggingface.co/v1/production/uploads/643e5d6a1d0e956d94bb3608/mTnu7Dts2RmDklHlp9Gqu.jpeg
Xiaoyang Wu
user
Gofinge
67dd5935b4cce80ee64cd990
Daniel DeTone
admin_assigned
2025-03-21T13:09:34.106Z
66f1e84ea4d3e85a77bdb545
/avatars/4e24b0bdd7a8c86ba5f43f4b042817ea.svg
Daniel DeTone
user
ddetone
67dd5935b4cce80ee64cd991
Duncan Frost
admin_assigned
2025-03-21T13:09:54.292Z
65fc12c6fc9132a2dfe9e8b3
/avatars/3d04d79721f4ec6b0b090e0bbbf85f70.svg
Duncan Frost
user
frosd01
67dd5935b4cce80ee64cd992
Tianwei Shen
admin_assigned
2025-03-21T13:10:00.797Z
66e0950be1bfac4b463fb941
/avatars/6506d169af3ead9b7d7d30a1013857b1.svg
Tianwei Shen
user
tshenaa
67dd5935b4cce80ee64cd993
Chris Xie
67dd5935b4cce80ee64cd994
Nan Yang
67dd5935b4cce80ee64cd995
Jakob Engel
admin_assigned
2025-03-21T13:10:15.646Z
62861d244713da0266f29f99
/avatars/819abb2f7d02d65cbaa4a571b8b13169.svg
Jakob Engel
user
JakobEngel
67dd5935b4cce80ee64cd996
Richard Newcombe
admin_assigned
2025-03-21T13:10:22.340Z
633ea8814b469e3151bb38f2
/avatars/245f849cde665f1d7dfec15dead6b880.svg
Richard Newcombe
user
MONGIE30
67dd5935b4cce80ee64cd997
Hengshuang Zhao
67dd5935b4cce80ee64cd998
Julian Straub
admin_assigned
2025-03-21T13:10:37.008Z
673927a0ab4212713230d456
/avatars/b90972cefa847649a6acc47e1f100b3b.svg
Julian Straub
user
jstraub
Sonata: Self-Supervised Learning of Reliable Point Representations
In this paper, we question whether we have a reliable self-supervised point
cloud model that can be used for diverse 3D tasks via simple linear probing,
even with limited data and minimal computation. We find that existing 3D
self-supervised learning approaches fall short when evaluated on representation
quality through linear probing. We hypothesize that this is due to what we term
the "geometric shortcut", which causes representations to collapse to low-level
spatial features. This challenge is unique to 3D and arises from the sparse
nature of point cloud data. We address it through two key strategies: obscuring
spatial information and enhancing the reliance on input features, ultimately
composing a Sonata of 140k point clouds through self-distillation. Sonata is
simple and intuitive, yet its learned representations are strong and reliable:
zero-shot visualizations demonstrate semantic grouping, alongside strong
spatial reasoning through nearest-neighbor relationships. Sonata demonstrates
exceptional parameter and data efficiency, tripling linear probing accuracy
(from 21.8% to 72.5%) on ScanNet and nearly doubling performance with only 1%
of the data compared to previous approaches. Full fine-tuning further advances
SOTA across both 3D indoor and outdoor perception tasks.
The paper investigates the effectiveness of self-supervised 3D point cloud models, addressing the "geometric shortcut" issue through spatial obscuring and feature enhancement, resulting in strong parameter and data efficient representations.
self-supervised learning point cloud data linear probing geometric shortcut spatial features self-distillation nearest-neighbor relationships ScanNet 3D indoor perception 3D outdoor perception