ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their
  Self-Supervised Pretraining
3D Gaussian Splatting (3DGS) has become the de facto method of 3D
representation in many vision tasks. This calls for the 3D understanding
directly in this representation space. To facilitate the research in this
direction, we first build a large-scale dataset of 3DGS using the commonly used
ShapeNet and ModelNet datasets. Our dataset ShapeSplat consists of 65K objects
from 87 unique categories, whose labels are in accordance with the respective
datasets. The creation of this dataset utilized the compute equivalent of 2 GPU
years on a TITAN XP GPU.
  We utilize our dataset for unsupervised pretraining and supervised finetuning
for classification and segmentation tasks. To this end, we introduce
\textit{Gaussian-MAE}, which highlights the unique benefits of
representation learning from Gaussian parameters. Through exhaustive
experiments, we provide several valuable insights. In particular, we show that
(1) the distribution of the optimized GS centroids significantly differs from
the uniformly sampled point cloud (used for initialization) counterpart; (2)
this change in distribution results in degradation in classification but
improvement in segmentation tasks when using only the centroids; (3) to
leverage additional Gaussian parameters, we propose Gaussian feature grouping
in a normalized feature space, along with splats pooling layer, offering a
tailored solution to effectively group and embed similar Gaussians, which leads
to notable improvement in finetuning tasks.
66c55822e9d46e06d5d4ca88
Qi Ma
66c55822e9d46e06d5d4ca89
Yue Li
66c55822e9d46e06d5d4ca8a
Bin Ren
66c55822e9d46e06d5d4ca8b
Nicu Sebe
66c55822e9d46e06d5d4ca8c
Ender Konukoglu
66c55822e9d46e06d5d4ca8d
Theo Gevers
66c55822e9d46e06d5d4ca8e
Luc Van Gool
66c55822e9d46e06d5d4ca8f
Danda Pani Paudel
ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their
  Self-Supervised Pretraining
3D Gaussian Splatting (3DGS) has become the de facto method of 3D
representation in many vision tasks. This calls for the 3D understanding
directly in this representation space. To facilitate the research in this
direction, we first build a large-scale dataset of 3DGS using the commonly used
ShapeNet and ModelNet datasets. Our dataset ShapeSplat consists of 65K objects
from 87 unique categories, whose labels are in accordance with the respective
datasets. The creation of this dataset utilized the compute equivalent of 2 GPU
years on a TITAN XP GPU.
  We utilize our dataset for unsupervised pretraining and supervised finetuning
for classification and segmentation tasks. To this end, we introduce
\textit{Gaussian-MAE}, which highlights the unique benefits of
representation learning from Gaussian parameters. Through exhaustive
experiments, we provide several valuable insights. In particular, we show that
(1) the distribution of the optimized GS centroids significantly differs from
the uniformly sampled point cloud (used for initialization) counterpart; (2)
this change in distribution results in degradation in classification but
improvement in segmentation tasks when using only the centroids; (3) to
leverage additional Gaussian parameters, we propose Gaussian feature grouping
in a normalized feature space, along with splats pooling layer, offering a
tailored solution to effectively group and embed similar Gaussians, which leads
to notable improvement in finetuning tasks.
A large dataset of 3D Gaussian Splatting is created to facilitate unsupervised pretraining and supervised finetuning for 3D classification and segmentation tasks, introducing Gaussian-MAE and Gaussian feature grouping techniques.
3D Gaussian Splatting Gaussian-MAE Gaussian feature grouping splats pooling layer