Improving 2D Feature Representations by 3D-Aware Fine-Tuning
Current visual foundation models are trained purely on unstructured 2D data,
limiting their understanding of 3D structure of objects and scenes. In this
work, we show that fine-tuning on 3D-aware data improves the quality of
emerging semantic features. We design a method to lift semantic 2D features
into an efficient 3D Gaussian representation, which allows us to re-render them
for arbitrary views. Using the rendered 3D-aware features, we design a
fine-tuning strategy to transfer such 3D awareness into a 2D foundation model.
We demonstrate that models fine-tuned in that way produce features that readily
improve downstream task performance in semantic segmentation and depth
estimation through simple linear probing. Notably, though fined-tuned on a
single indoor dataset, the improvement is transferable to a variety of indoor
datasets and out-of-domain datasets. We hope our study encourages the community
to consider injecting 3D awareness when training 2D foundation models. Project
page: https://ywyue.github.io/FiT3D.
66a854d99189298acbf49941
Yuanwen Yue
claimed_verified
2024-07-30T07:25:49.460Z
65b0b33a403a23a2fda96cbc
https://cdn-avatars.huggingface.co/v1/production/uploads/65b0b33a403a23a2fda96cbc/f-45UN20BixueeJPAuJUj.png
Yuanwen Yue
user
yuanwenyue
66a854d99189298acbf49942
Anurag Das
66a854d99189298acbf49943
Francis Engelmann
claimed_verified
2025-05-18T08:48:31.971Z
671a7160e45d92e304e82f45
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-trHIg97qX0dE9T4UYycc.png
Francis Engelmann
user
francisengelmann
66a854d99189298acbf49944
Siyu Tang
66a854d99189298acbf49945
Jan Eric Lenssen
Improving 2D Feature Representations by 3D-Aware Fine-Tuning
Current visual foundation models are trained purely on unstructured 2D data,
limiting their understanding of 3D structure of objects and scenes. In this
work, we show that fine-tuning on 3D-aware data improves the quality of
emerging semantic features. We design a method to lift semantic 2D features
into an efficient 3D Gaussian representation, which allows us to re-render them
for arbitrary views. Using the rendered 3D-aware features, we design a
fine-tuning strategy to transfer such 3D awareness into a 2D foundation model.
We demonstrate that models fine-tuned in that way produce features that readily
improve downstream task performance in semantic segmentation and depth
estimation through simple linear probing. Notably, though fined-tuned on a
single indoor dataset, the improvement is transferable to a variety of indoor
datasets and out-of-domain datasets. We hope our study encourages the community
to consider injecting 3D awareness when training 2D foundation models. Project
page: https://ywyue.github.io/FiT3D.
Injecting 3D awareness by lifting 2D features into a 3D Gaussian representation enhances semantic understanding and downstream task performance in 2D foundation models.
Gaussian representation 3D Gaussian representation semantic segmentation depth estimation linear probing