Multiview Equivariance Improves 3D Correspondence Understanding with
  Minimal Feature Finetuning
Vision foundation models, particularly the ViT family, have revolutionized
image understanding by providing rich semantic features. However, despite their
success in 2D comprehension, their abilities on grasping 3D spatial
relationships are still unclear. In this work, we evaluate and enhance the 3D
awareness of ViT-based models. We begin by systematically assessing their
ability to learn 3D equivariant features, specifically examining the
consistency of semantic embeddings across different viewpoints. Our findings
indicate that improved 3D equivariance leads to better performance on various
downstream tasks, including pose estimation, tracking, and semantic transfer.
Building on this insight, we propose a simple yet effective finetuning strategy
based on 3D correspondences, which significantly enhances the 3D correspondence
understanding of existing vision models. Remarkably, even finetuning on a
single object for just one iteration results in substantial performance gains.
All code and resources will be made publicly available to support further
advancements in 3D-aware vision models. Our code is available at
https://github.com/qq456cvb/3DCorrEnhance.
6796e43ce05ca91d7eb430b5
Yang You
claimed_verified
2025-01-27T10:43:22.673Z
64791d760c2c2297c335fb24
/avatars/716c3c2ea42baf074ba9680c9939da28.svg
Yang You
user
qq456cvb
6796e43ce05ca91d7eb430b6
Yixin Li
admin_assigned
2025-01-27T15:10:55.826Z
66315569e9960a198f7cf634
/avatars/9fc7dd96a3ba1e47d9574c226e0101e0.svg
Yixin Li
user
yixinli
6796e43ce05ca91d7eb430b7
Congyue Deng
admin_assigned
2025-01-27T15:11:05.862Z
634483306e988c6a792d9a9b
/avatars/da717bf4e270ec0e2c8d8a82f8884081.svg
Deng
user
Congyue
6796e43ce05ca91d7eb430b8
Yue Wang
6796e43ce05ca91d7eb430b9
Leonidas Guibas
Multiview Equivariance Improves 3D Correspondence Understanding with
  Minimal Feature Finetuning
Vision foundation models, particularly the ViT family, have revolutionized
image understanding by providing rich semantic features. However, despite their
success in 2D comprehension, their abilities on grasping 3D spatial
relationships are still unclear. In this work, we evaluate and enhance the 3D
awareness of ViT-based models. We begin by systematically assessing their
ability to learn 3D equivariant features, specifically examining the
consistency of semantic embeddings across different viewpoints. Our findings
indicate that improved 3D equivariance leads to better performance on various
downstream tasks, including pose estimation, tracking, and semantic transfer.
Building on this insight, we propose a simple yet effective finetuning strategy
based on 3D correspondences, which significantly enhances the 3D correspondence
understanding of existing vision models. Remarkably, even finetuning on a
single object for just one iteration results in substantial performance gains.
All code and resources will be made publicly available to support further
advancements in 3D-aware vision models. Our code is available at
https://github.com/qq456cvb/3DCorrEnhance.
Evaluating and enhancing 3D equivariance in ViT-based models improves performance in tasks like pose estimation, tracking, and semantic transfer through simple fine-tuning with 3D correspondences.
ViT family 3D equivariant features semantic embeddings pose estimation tracking semantic transfer finetuning strategy 3D correspondences