title: ODIN: A Single Model for 2D and 3D Perception
summary: State-of-the-art models on contemporary 3D perception benchmarks like ScanNet
  consume and label dataset-provided 3D point clouds, obtained through post
  processing of sensed multiview RGB-D images. They are typically trained
  in-domain, forego large-scale 2D pre-training and outperform alternatives that
  featurize the posed RGB-D multiview images instead. The gap in performance
  between methods that consume posed images versus post-processed 3D point clouds
  has fueled the belief that 2D and 3D perception require distinct model
  architectures. In this paper, we challenge this view and propose ODIN
  (Omni-Dimensional INstance segmentation), a model that can segment and label
  both 2D RGB images and 3D point clouds, using a transformer architecture that
  alternates between 2D within-view and 3D cross-view information fusion. Our
  model differentiates 2D and 3D feature operations through the positional
  encodings of the tokens involved, which capture pixel coordinates for 2D patch
  tokens and 3D coordinates for 3D feature tokens. ODIN achieves state-of-the-art
  performance on ScanNet200, Matterport3D and AI2THOR 3D instance segmentation
  benchmarks, and competitive performance on ScanNet, S3DIS and COCO. It
  outperforms all previous works by a wide margin when the sensed 3D point cloud
  is used in place of the point cloud sampled from 3D mesh. When used as the 3D
  perception engine in an instructable embodied agent architecture, it sets a new
  state-of-the-art on the TEACh action-from-dialogue benchmark. Our code and
  checkpoints can be found at the project website: https://odin-seg.github.io.
paper_authors._id: 659774c41207310bb88feeb3
paper_authors.name: Ayush Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-05T17:18:43.296Z
paper_authors._id: 5ff57866690a009add03cf2b
paper_authors.avatarUrl: /avatars/6ac16938543b3ad47235c00529b4b031.svg
paper_authors.fullname: Ayush Jain
paper_authors.type: user
paper_authors.user: ayushjain1144
paper_authors._id: 659774c41207310bb88feeb4
paper_authors.name: Pushkal Katara
paper_authors._id: 659774c41207310bb88feeb5
paper_authors.name: Nikolaos Gkanatsios
paper_authors._id: 659774c41207310bb88feeb6
paper_authors.name: Adam W. Harley
paper_authors._id: 659774c41207310bb88feeb7
paper_authors.name: Gabriel Sarch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T18:51:59.113Z
paper_authors._id: 6666828b15ec3fd28b52a1bd
paper_authors.avatarUrl: /avatars/a80eeac5dd23aabce1eb299ebe7f245b.svg
paper_authors.fullname: Gabriel H Sarch
paper_authors.type: user
paper_authors.user: gsarch
paper_authors._id: 659774c41207310bb88feeb8
paper_authors.name: Kriti Aggarwal
paper_authors._id: 659774c41207310bb88feeb9
paper_authors.name: Vishrav Chaudhary
paper_authors._id: 659774c41207310bb88feeba
paper_authors.name: Katerina Fragkiadaki
paper_title: ODIN: A Single Model for 2D and 3D Perception
paper_summary: State-of-the-art models on contemporary 3D perception benchmarks like ScanNet
  consume and label dataset-provided 3D point clouds, obtained through post
  processing of sensed multiview RGB-D images. They are typically trained
  in-domain, forego large-scale 2D pre-training and outperform alternatives that
  featurize the posed RGB-D multiview images instead. The gap in performance
  between methods that consume posed images versus post-processed 3D point clouds
  has fueled the belief that 2D and 3D perception require distinct model
  architectures. In this paper, we challenge this view and propose ODIN
  (Omni-Dimensional INstance segmentation), a model that can segment and label
  both 2D RGB images and 3D point clouds, using a transformer architecture that
  alternates between 2D within-view and 3D cross-view information fusion. Our
  model differentiates 2D and 3D feature operations through the positional
  encodings of the tokens involved, which capture pixel coordinates for 2D patch
  tokens and 3D coordinates for 3D feature tokens. ODIN achieves state-of-the-art
  performance on ScanNet200, Matterport3D and AI2THOR 3D instance segmentation
  benchmarks, and competitive performance on ScanNet, S3DIS and COCO. It
  outperforms all previous works by a wide margin when the sensed 3D point cloud
  is used in place of the point cloud sampled from 3D mesh. When used as the 3D
  perception engine in an instructable embodied agent architecture, it sets a new
  state-of-the-art on the TEACh action-from-dialogue benchmark. Our code and
  checkpoints can be found at the project website: https://odin-seg.github.io.
paper_ai_keywords: transformer architecture positional encodings 2D patch tokens 3D feature tokens