title: 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features
summary: We present 3DiffTection, a state-of-the-art method for 3D object detection
  from single images, leveraging features from a 3D-aware diffusion model.
  Annotating large-scale image data for 3D detection is resource-intensive and
  time-consuming. Recently, pretrained large image diffusion models have become
  prominent as effective feature extractors for 2D perception tasks. However,
  these features are initially trained on paired text and image data, which are
  not optimized for 3D tasks, and often exhibit a domain gap when applied to the
  target data. Our approach bridges these gaps through two specialized tuning
  strategies: geometric and semantic. For geometric tuning, we fine-tune a
  diffusion model to perform novel view synthesis conditioned on a single image,
  by introducing a novel epipolar warp operator. This task meets two essential
  criteria: the necessity for 3D awareness and reliance solely on posed image
  data, which are readily available (e.g., from videos) and does not require
  manual annotation. For semantic refinement, we further train the model on
  target data with detection supervision. Both tuning phases employ ControlNet to
  preserve the integrity of the original feature capabilities. In the final step,
  we harness these enhanced capabilities to conduct a test-time prediction
  ensemble across multiple virtual viewpoints. Through our methodology, we obtain
  3D-aware features that are tailored for 3D detection and excel in identifying
  cross-view point correspondences. Consequently, our model emerges as a powerful
  3D detector, substantially surpassing previous benchmarks, e.g., Cube-RCNN, a
  precedent in single-view 3D detection by 9.43\% in AP3D on the
  Omni3D-ARkitscene dataset. Furthermore, 3DiffTection showcases robust data
  efficiency and generalization to cross-domain data.
paper_authors._id: 654c5d70386fc5525c197a13
paper_authors.name: Chenfeng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:57:14.754Z
paper_authors._id: 654ca71d5255ee86711b52c5
paper_authors.avatarUrl: /avatars/52bf00fd74c8db5643c4daa185c678e6.svg
paper_authors.fullname: Chenfeng Xu
paper_authors.type: user
paper_authors.user: chenfengx
paper_authors._id: 654c5d70386fc5525c197a14
paper_authors.name: Huan Ling
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:09:22.807Z
paper_authors._id: 6344bf36762379fc6302abed
paper_authors.avatarUrl: /avatars/f4442fef316ac96652d4f99238822209.svg
paper_authors.fullname: huanling
paper_authors.type: user
paper_authors.user: huanling
paper_authors._id: 654c5d70386fc5525c197a15
paper_authors.name: Sanja Fidler
paper_authors._id: 654c5d70386fc5525c197a16
paper_authors.name: Or Litany
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:09:35.381Z
paper_authors._id: 630509558d7f39c0ecf29dc3
paper_authors.avatarUrl: /avatars/6b225b95f625152dc69fedb5f0b348d6.svg
paper_authors.fullname: Or Litany
paper_authors.type: user
paper_authors.user: orlitany
paper_title: 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features
paper_summary: We present 3DiffTection, a state-of-the-art method for 3D object detection
  from single images, leveraging features from a 3D-aware diffusion model.
  Annotating large-scale image data for 3D detection is resource-intensive and
  time-consuming. Recently, pretrained large image diffusion models have become
  prominent as effective feature extractors for 2D perception tasks. However,
  these features are initially trained on paired text and image data, which are
  not optimized for 3D tasks, and often exhibit a domain gap when applied to the
  target data. Our approach bridges these gaps through two specialized tuning
  strategies: geometric and semantic. For geometric tuning, we fine-tune a
  diffusion model to perform novel view synthesis conditioned on a single image,
  by introducing a novel epipolar warp operator. This task meets two essential
  criteria: the necessity for 3D awareness and reliance solely on posed image
  data, which are readily available (e.g., from videos) and does not require
  manual annotation. For semantic refinement, we further train the model on
  target data with detection supervision. Both tuning phases employ ControlNet to
  preserve the integrity of the original feature capabilities. In the final step,
  we harness these enhanced capabilities to conduct a test-time prediction
  ensemble across multiple virtual viewpoints. Through our methodology, we obtain
  3D-aware features that are tailored for 3D detection and excel in identifying
  cross-view point correspondences. Consequently, our model emerges as a powerful
  3D detector, substantially surpassing previous benchmarks, e.g., Cube-RCNN, a
  precedent in single-view 3D detection by 9.43\% in AP3D on the
  Omni3D-ARkitscene dataset. Furthermore, 3DiffTection showcases robust data
  efficiency and generalization to cross-domain data.
paper_ai_summary: 3DiffTection enhances 3D object detection from single images using a 3D-aware diffusion model with specialized tuning strategies, achieving state-of-the-art performance and robust generalization.
paper_ai_keywords: 3D object detection 3DiffTection diffusion model 3D-aware novel view synthesis epipolar warp operator semantic refinement ControlNet ensemble prediction 3D detection features cross-view point correspondences data efficiency generalized performance