title: ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their
    Self-Supervised Pretraining
summary: 3D Gaussian Splatting (3DGS) has become the de facto method of 3D
  representation in many vision tasks. This calls for the 3D understanding
  directly in this representation space. To facilitate the research in this
  direction, we first build a large-scale dataset of 3DGS using the commonly used
  ShapeNet and ModelNet datasets. Our dataset ShapeSplat consists of 65K objects
  from 87 unique categories, whose labels are in accordance with the respective
  datasets. The creation of this dataset utilized the compute equivalent of 2 GPU
  years on a TITAN XP GPU.
    We utilize our dataset for unsupervised pretraining and supervised finetuning
  for classification and segmentation tasks. To this end, we introduce
  \textit{Gaussian-MAE}, which highlights the unique benefits of
  representation learning from Gaussian parameters. Through exhaustive
  experiments, we provide several valuable insights. In particular, we show that
  (1) the distribution of the optimized GS centroids significantly differs from
  the uniformly sampled point cloud (used for initialization) counterpart; (2)
  this change in distribution results in degradation in classification but
  improvement in segmentation tasks when using only the centroids; (3) to
  leverage additional Gaussian parameters, we propose Gaussian feature grouping
  in a normalized feature space, along with splats pooling layer, offering a
  tailored solution to effectively group and embed similar Gaussians, which leads
  to notable improvement in finetuning tasks.
paper_authors._id: 66c55822e9d46e06d5d4ca88
paper_authors.name: Qi Ma
paper_authors._id: 66c55822e9d46e06d5d4ca89
paper_authors.name: Yue Li
paper_authors._id: 66c55822e9d46e06d5d4ca8a
paper_authors.name: Bin Ren
paper_authors._id: 66c55822e9d46e06d5d4ca8b
paper_authors.name: Nicu Sebe
paper_authors._id: 66c55822e9d46e06d5d4ca8c
paper_authors.name: Ender Konukoglu
paper_authors._id: 66c55822e9d46e06d5d4ca8d
paper_authors.name: Theo Gevers
paper_authors._id: 66c55822e9d46e06d5d4ca8e
paper_authors.name: Luc Van Gool
paper_authors._id: 66c55822e9d46e06d5d4ca8f
paper_authors.name: Danda Pani Paudel
paper_title: ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their
    Self-Supervised Pretraining
paper_summary: 3D Gaussian Splatting (3DGS) has become the de facto method of 3D
  representation in many vision tasks. This calls for the 3D understanding
  directly in this representation space. To facilitate the research in this
  direction, we first build a large-scale dataset of 3DGS using the commonly used
  ShapeNet and ModelNet datasets. Our dataset ShapeSplat consists of 65K objects
  from 87 unique categories, whose labels are in accordance with the respective
  datasets. The creation of this dataset utilized the compute equivalent of 2 GPU
  years on a TITAN XP GPU.
    We utilize our dataset for unsupervised pretraining and supervised finetuning
  for classification and segmentation tasks. To this end, we introduce
  \textit{Gaussian-MAE}, which highlights the unique benefits of
  representation learning from Gaussian parameters. Through exhaustive
  experiments, we provide several valuable insights. In particular, we show that
  (1) the distribution of the optimized GS centroids significantly differs from
  the uniformly sampled point cloud (used for initialization) counterpart; (2)
  this change in distribution results in degradation in classification but
  improvement in segmentation tasks when using only the centroids; (3) to
  leverage additional Gaussian parameters, we propose Gaussian feature grouping
  in a normalized feature space, along with splats pooling layer, offering a
  tailored solution to effectively group and embed similar Gaussians, which leads
  to notable improvement in finetuning tasks.
paper_ai_keywords: 3D Gaussian Splatting (3DGS) Gaussian-MAE ShapeSchematic Gaussian parameters GS centroids point cloud Gaussian feature grouping normalized feature space splats pooling layer