title: Probing the 3D Awareness of Visual Foundation Models
summary: Recent advances in large-scale pretraining have yielded visual foundation
  models with strong capabilities. Not only can recent models generalize to
  arbitrary images for their training task, their intermediate representations
  are useful for other visual tasks such as detection and segmentation. Given
  that such models can classify, delineate, and localize objects in 2D, we ask
  whether they also represent their 3D structure? In this work, we analyze the 3D
  awareness of visual foundation models. We posit that 3D awareness implies that
  representations (1) encode the 3D structure of the scene and (2) consistently
  represent the surface across views. We conduct a series of experiments using
  task-specific probes and zero-shot inference procedures on frozen features. Our
  experiments reveal several limitations of the current models. Our code and
  analysis can be found at https://github.com/mbanani/probe3d.
paper_authors._id: 661c8718c153e4a0a2a83ad6
paper_authors.name: Mohamed El Banani
paper_authors._id: 661c8718c153e4a0a2a83ad7
paper_authors.name: Amit Raj
paper_authors._id: 661c8718c153e4a0a2a83ad8
paper_authors.name: Kevis-Kokitsi Maninis
paper_authors._id: 661c8718c153e4a0a2a83ad9
paper_authors.name: Abhishek Kar
paper_authors._id: 661c8718c153e4a0a2a83ada
paper_authors.name: Yuanzhen Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:43:50.210Z
paper_authors._id: 6334ee5831a2be3938c85639
paper_authors.avatarUrl: /avatars/9821af26a585f54b595ec39f061f2250.svg
paper_authors.fullname: Yuanzhen Li
paper_authors.type: user
paper_authors.user: yzli
paper_authors._id: 661c8718c153e4a0a2a83adb
paper_authors.name: Michael Rubinstein
paper_authors._id: 661c8718c153e4a0a2a83adc
paper_authors.name: Deqing Sun
paper_authors._id: 661c8718c153e4a0a2a83add
paper_authors.name: Leonidas Guibas
paper_authors._id: 661c8718c153e4a0a2a83ade
paper_authors.name: Justin Johnson
paper_authors._id: 661c8718c153e4a0a2a83adf
paper_authors.name: Varun Jampani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:43:31.148Z
paper_authors._id: 630e7cfe3fc17ffc50f6e835
paper_authors.avatarUrl: /avatars/0b742ff094a09f9374fafcd97ab9e002.svg
paper_authors.fullname: Varun Jampani
paper_authors.type: user
paper_authors.user: varunjampani
paper_title: Probing the 3D Awareness of Visual Foundation Models
paper_summary: Recent advances in large-scale pretraining have yielded visual foundation
  models with strong capabilities. Not only can recent models generalize to
  arbitrary images for their training task, their intermediate representations
  are useful for other visual tasks such as detection and segmentation. Given
  that such models can classify, delineate, and localize objects in 2D, we ask
  whether they also represent their 3D structure? In this work, we analyze the 3D
  awareness of visual foundation models. We posit that 3D awareness implies that
  representations (1) encode the 3D structure of the scene and (2) consistently
  represent the surface across views. We conduct a series of experiments using
  task-specific probes and zero-shot inference procedures on frozen features. Our
  experiments reveal several limitations of the current models. Our code and
  analysis can be found at https://github.com/mbanani/probe3d.
paper_ai_summary: Visual foundation models are analyzed for their 3D awareness through task-specific probes and zero-shot inference on frozen features, revealing limitations.
paper_ai_keywords: 3D awareness representations scene 3D structure surface views task-specific probes zero-shot inference