title: Multiview Equivariance Improves 3D Correspondence Understanding with
    Minimal Feature Finetuning
summary: Vision foundation models, particularly the ViT family, have revolutionized
  image understanding by providing rich semantic features. However, despite their
  success in 2D comprehension, their abilities on grasping 3D spatial
  relationships are still unclear. In this work, we evaluate and enhance the 3D
  awareness of ViT-based models. We begin by systematically assessing their
  ability to learn 3D equivariant features, specifically examining the
  consistency of semantic embeddings across different viewpoints. Our findings
  indicate that improved 3D equivariance leads to better performance on various
  downstream tasks, including pose estimation, tracking, and semantic transfer.
  Building on this insight, we propose a simple yet effective finetuning strategy
  based on 3D correspondences, which significantly enhances the 3D correspondence
  understanding of existing vision models. Remarkably, even finetuning on a
  single object for just one iteration results in substantial performance gains.
  All code and resources will be made publicly available to support further
  advancements in 3D-aware vision models. Our code is available at
  https://github.com/qq456cvb/3DCorrEnhance.
paper_authors._id: 6796e43ce05ca91d7eb430b5
paper_authors.name: Yang You
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-27T10:43:22.673Z
paper_authors._id: 64791d760c2c2297c335fb24
paper_authors.avatarUrl: /avatars/716c3c2ea42baf074ba9680c9939da28.svg
paper_authors.fullname: Yang You
paper_authors.type: user
paper_authors.user: qq456cvb
paper_authors._id: 6796e43ce05ca91d7eb430b6
paper_authors.name: Yixin Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T15:10:55.826Z
paper_authors._id: 66315569e9960a198f7cf634
paper_authors.avatarUrl: /avatars/9fc7dd96a3ba1e47d9574c226e0101e0.svg
paper_authors.fullname: Yixin Li
paper_authors.type: user
paper_authors.user: yixinli
paper_authors._id: 6796e43ce05ca91d7eb430b7
paper_authors.name: Congyue Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T15:11:05.862Z
paper_authors._id: 634483306e988c6a792d9a9b
paper_authors.avatarUrl: /avatars/da717bf4e270ec0e2c8d8a82f8884081.svg
paper_authors.fullname: Deng
paper_authors.type: user
paper_authors.user: Congyue
paper_authors._id: 6796e43ce05ca91d7eb430b8
paper_authors.name: Yue Wang
paper_authors._id: 6796e43ce05ca91d7eb430b9
paper_authors.name: Leonidas Guibas
paper_title: Multiview Equivariance Improves 3D Correspondence Understanding with
    Minimal Feature Finetuning
paper_summary: Vision foundation models, particularly the ViT family, have revolutionized
  image understanding by providing rich semantic features. However, despite their
  success in 2D comprehension, their abilities on grasping 3D spatial
  relationships are still unclear. In this work, we evaluate and enhance the 3D
  awareness of ViT-based models. We begin by systematically assessing their
  ability to learn 3D equivariant features, specifically examining the
  consistency of semantic embeddings across different viewpoints. Our findings
  indicate that improved 3D equivariance leads to better performance on various
  downstream tasks, including pose estimation, tracking, and semantic transfer.
  Building on this insight, we propose a simple yet effective finetuning strategy
  based on 3D correspondences, which significantly enhances the 3D correspondence
  understanding of existing vision models. Remarkably, even finetuning on a
  single object for just one iteration results in substantial performance gains.
  All code and resources will be made publicly available to support further
  advancements in 3D-aware vision models. Our code is available at
  https://github.com/qq456cvb/3DCorrEnhance.
paper_ai_keywords: ViT family image understanding semantic features 3D spatial relationships 3D equivariant features semantic embeddings pose estimation tracking semantic transfer finetuning strategy 3D correspondences 3D-aware vision models