title: CrossOver: 3D Scene Cross-Modal Alignment
summary: Multi-modal 3D object understanding has gained significant attention, yet
  current approaches often assume complete data availability and rigid alignment
  across all modalities. We present CrossOver, a novel framework for cross-modal
  3D scene understanding via flexible, scene-level modality alignment. Unlike
  traditional methods that require aligned modality data for every object
  instance, CrossOver learns a unified, modality-agnostic embedding space for
  scenes by aligning modalities - RGB images, point clouds, CAD models,
  floorplans, and text descriptions - with relaxed constraints and without
  explicit object semantics. Leveraging dimensionality-specific encoders, a
  multi-stage training pipeline, and emergent cross-modal behaviors, CrossOver
  supports robust scene retrieval and object localization, even with missing
  modalities. Evaluations on ScanNet and 3RScan datasets show its superior
  performance across diverse metrics, highlighting adaptability for real-world
  applications in 3D scene understanding.
paper_authors._id: 67bc0d12ffc2c387329c8cfd
paper_authors.name: Sayan Deb Sarkar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:06:56.555Z
paper_authors._id: 650ec19e6620b0c57e2a551b
paper_authors.avatarUrl: /avatars/c26c03fa920d857120f03c9ccb9f1d7a.svg
paper_authors.fullname: Sayan Deb Sarkar
paper_authors.type: user
paper_authors.user: sayandsarkar
paper_authors._id: 67bc0d12ffc2c387329c8cfe
paper_authors.name: Ondrej Miksik
paper_authors._id: 67bc0d12ffc2c387329c8cff
paper_authors.name: Marc Pollefeys
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T16:11:04.481Z
paper_authors._id: 67b5fa179782a5e2fd2cb26a
paper_authors.avatarUrl: /avatars/62c38f29ec641e001eeddf840bea21a0.svg
paper_authors.fullname: Marc Pollefeys
paper_authors.type: user
paper_authors.user: mapo1
paper_authors._id: 67bc0d12ffc2c387329c8d00
paper_authors.name: Daniel Barath
paper_authors._id: 67bc0d12ffc2c387329c8d01
paper_authors.name: Iro Armeni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T16:11:14.554Z
paper_authors._id: 6745f90cf4d75fd11a2407ac
paper_authors.avatarUrl: /avatars/882f56565b4ebfabf1c13e199d74a4de.svg
paper_authors.fullname: Iro Armeni
paper_authors.type: user
paper_authors.user: ir0
paper_title: CrossOver: 3D Scene Cross-Modal Alignment
paper_summary: Multi-modal 3D object understanding has gained significant attention, yet
  current approaches often assume complete data availability and rigid alignment
  across all modalities. We present CrossOver, a novel framework for cross-modal
  3D scene understanding via flexible, scene-level modality alignment. Unlike
  traditional methods that require aligned modality data for every object
  instance, CrossOver learns a unified, modality-agnostic embedding space for
  scenes by aligning modalities - RGB images, point clouds, CAD models,
  floorplans, and text descriptions - with relaxed constraints and without
  explicit object semantics. Leveraging dimensionality-specific encoders, a
  multi-stage training pipeline, and emergent cross-modal behaviors, CrossOver
  supports robust scene retrieval and object localization, even with missing
  modalities. Evaluations on ScanNet and 3RScan datasets show its superior
  performance across diverse metrics, highlighting adaptability for real-world
  applications in 3D scene understanding.
paper_ai_keywords: CrossOver 3D scene understanding modality alignment unified embedding space RGB images point clouds CAD models floorplans text descriptions dimensionality-specific encoders multi-stage training pipeline emergent cross-modal behaviors scene retrieval object localization