title: Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent
    Spaces
summary: Recent works have shown that, when trained at scale, uni-modal 2D vision and
  text encoders converge to learned features that share remarkable structural
  properties, despite arising from different representations. However, the role
  of 3D encoders with respect to other modalities remains unexplored.
  Furthermore, existing 3D foundation models that leverage large datasets are
  typically trained with explicit alignment objectives with respect to frozen
  encoders from other representations. In this work, we investigate the
  possibility of a posteriori alignment of representations obtained from
  uni-modal 3D encoders compared to text-based feature spaces. We show that naive
  post-training feature alignment of uni-modal text and 3D encoders results in
  limited performance. We then focus on extracting subspaces of the corresponding
  feature spaces and discover that by projecting learned representations onto
  well-chosen lower-dimensional subspaces the quality of alignment becomes
  significantly higher, leading to improved accuracy on matching and retrieval
  tasks. Our analysis further sheds light on the nature of these shared
  subspaces, which roughly separate between semantic and geometric data
  representations. Overall, ours is the first work that helps to establish a
  baseline for post-training alignment of 3D uni-modal and text feature spaces,
  and helps to highlight both the shared and unique properties of 3D data
  compared to other representations.
paper_authors._id: 67cef721e5ab8ec0550b7a66
paper_authors.name: Souhail Hadgi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:18:09.155Z
paper_authors._id: 6542cb9602409f0994fbb6c6
paper_authors.avatarUrl: /avatars/c0ba12b349e61998f36f2701ac3acb30.svg
paper_authors.fullname: Souhail Hadgi
paper_authors.type: user
paper_authors.user: Squall01
paper_authors._id: 67cef721e5ab8ec0550b7a67
paper_authors.name: Luca Moschella
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:18:16.141Z
paper_authors._id: 607021f450a44430c625a937
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1617961457504-noauth.jpeg
paper_authors.fullname: Luca Moschella
paper_authors.type: user
paper_authors.user: lucmos
paper_authors._id: 67cef721e5ab8ec0550b7a68
paper_authors.name: Andrea Santilli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T14:35:44.397Z
paper_authors._id: 5e8ef1f14957053f606489e6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1635502086699-5e8ef1f14957053f606489e6.jpeg
paper_authors.fullname: Andrea Santilli
paper_authors.type: user
paper_authors.user: teelinsan
paper_authors._id: 67cef721e5ab8ec0550b7a69
paper_authors.name: Diego Gomez
paper_authors._id: 67cef721e5ab8ec0550b7a6a
paper_authors.name: Qixing Huang
paper_authors._id: 67cef721e5ab8ec0550b7a6b
paper_authors.name: Emanuele Rodolà
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:18:47.736Z
paper_authors._id: 652681664e066bf73f8e2bd1
paper_authors.avatarUrl: /avatars/084dec4765d9996d74901b8df95ec35f.svg
paper_authors.fullname: Emanuele Rodola'
paper_authors.type: user
paper_authors.user: erodola
paper_authors._id: 67cef721e5ab8ec0550b7a6c
paper_authors.name: Simone Melzi
paper_authors._id: 67cef721e5ab8ec0550b7a6d
paper_authors.name: Maks Ovsjanikov
paper_title: Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent
    Spaces
paper_summary: Recent works have shown that, when trained at scale, uni-modal 2D vision and
  text encoders converge to learned features that share remarkable structural
  properties, despite arising from different representations. However, the role
  of 3D encoders with respect to other modalities remains unexplored.
  Furthermore, existing 3D foundation models that leverage large datasets are
  typically trained with explicit alignment objectives with respect to frozen
  encoders from other representations. In this work, we investigate the
  possibility of a posteriori alignment of representations obtained from
  uni-modal 3D encoders compared to text-based feature spaces. We show that naive
  post-training feature alignment of uni-modal text and 3D encoders results in
  limited performance. We then focus on extracting subspaces of the corresponding
  feature spaces and discover that by projecting learned representations onto
  well-chosen lower-dimensional subspaces the quality of alignment becomes
  significantly higher, leading to improved accuracy on matching and retrieval
  tasks. Our analysis further sheds light on the nature of these shared
  subspaces, which roughly separate between semantic and geometric data
  representations. Overall, ours is the first work that helps to establish a
  baseline for post-training alignment of 3D uni-modal and text feature spaces,
  and helps to highlight both the shared and unique properties of 3D data
  compared to other representations.
paper_ai_keywords: uni-modal 2D vision text encoders learned features 3D encoders 3D foundation models alignment objectives feature alignment subspaces lower-dimensional subspaces semantic data representations geometric data representations matching tasks retrieval tasks