ODIN: A Single Model for 2D and 3D Perception
State-of-the-art models on contemporary 3D perception benchmarks like ScanNet
consume and label dataset-provided 3D point clouds, obtained through post
processing of sensed multiview RGB-D images. They are typically trained
in-domain, forego large-scale 2D pre-training and outperform alternatives that
featurize the posed RGB-D multiview images instead. The gap in performance
between methods that consume posed images versus post-processed 3D point clouds
has fueled the belief that 2D and 3D perception require distinct model
architectures. In this paper, we challenge this view and propose ODIN
(Omni-Dimensional INstance segmentation), a model that can segment and label
both 2D RGB images and 3D point clouds, using a transformer architecture that
alternates between 2D within-view and 3D cross-view information fusion. Our
model differentiates 2D and 3D feature operations through the positional
encodings of the tokens involved, which capture pixel coordinates for 2D patch
tokens and 3D coordinates for 3D feature tokens. ODIN achieves state-of-the-art
performance on ScanNet200, Matterport3D and AI2THOR 3D instance segmentation
benchmarks, and competitive performance on ScanNet, S3DIS and COCO. It
outperforms all previous works by a wide margin when the sensed 3D point cloud
is used in place of the point cloud sampled from 3D mesh. When used as the 3D
perception engine in an instructable embodied agent architecture, it sets a new
state-of-the-art on the TEACh action-from-dialogue benchmark. Our code and
checkpoints can be found at the project website: https://odin-seg.github.io.
659774c41207310bb88feeb3
Ayush Jain
claimed_verified
2024-01-05T17:18:43.296Z
5ff57866690a009add03cf2b
/avatars/6ac16938543b3ad47235c00529b4b031.svg
Ayush Jain
user
ayushjain1144
659774c41207310bb88feeb4
Pushkal Katara
659774c41207310bb88feeb5
Nikolaos Gkanatsios
659774c41207310bb88feeb6
Adam W. Harley
claimed_verified
2025-07-02T07:48:32.127Z
6314ff478be84e1ab412155a
/avatars/5a0c0aedc99aac6731e0652faf98b2e6.svg
Adam W Harley
user
aharley
659774c41207310bb88feeb7
Gabriel Sarch
admin_assigned
2024-06-24T18:51:59.113Z
6666828b15ec3fd28b52a1bd
https://cdn-avatars.huggingface.co/v1/production/uploads/6666828b15ec3fd28b52a1bd/jRPAQmvg_J6J84Louar6P.jpeg
Gabriel H Sarch
user
gsarch
659774c41207310bb88feeb8
Kriti Aggarwal
659774c41207310bb88feeb9
Vishrav Chaudhary
659774c41207310bb88feeba
Katerina Fragkiadaki
ODIN: A Single Model for 2D and 3D Perception
State-of-the-art models on contemporary 3D perception benchmarks like ScanNet
consume and label dataset-provided 3D point clouds, obtained through post
processing of sensed multiview RGB-D images. They are typically trained
in-domain, forego large-scale 2D pre-training and outperform alternatives that
featurize the posed RGB-D multiview images instead. The gap in performance
between methods that consume posed images versus post-processed 3D point clouds
has fueled the belief that 2D and 3D perception require distinct model
architectures. In this paper, we challenge this view and propose ODIN
(Omni-Dimensional INstance segmentation), a model that can segment and label
both 2D RGB images and 3D point clouds, using a transformer architecture that
alternates between 2D within-view and 3D cross-view information fusion. Our
model differentiates 2D and 3D feature operations through the positional
encodings of the tokens involved, which capture pixel coordinates for 2D patch
tokens and 3D coordinates for 3D feature tokens. ODIN achieves state-of-the-art
performance on ScanNet200, Matterport3D and AI2THOR 3D instance segmentation
benchmarks, and competitive performance on ScanNet, S3DIS and COCO. It
outperforms all previous works by a wide margin when the sensed 3D point cloud
is used in place of the point cloud sampled from 3D mesh. When used as the 3D
perception engine in an instructable embodied agent architecture, it sets a new
state-of-the-art on the TEACh action-from-dialogue benchmark. Our code and
checkpoints can be found at the project website: https://odin-seg.github.io.
ODIN is a transformer-based model that segments both 2D and 3D data by alternating between 2D and 3D information fusion, achieving state-of-the-art performance on multiple benchmarks.
transformer architecture information fusion positional encodings tokens instance segmentation ScanNet200 Matterport3D AI2THOR ScanNet S3DIS COCO TEACh