Improving 2D Feature Representations by 3D-Aware Fine-Tuning
<|reference_start|>Improving 2D Feature Representations by 3D-Aware Fine-Tuning: Current visual foundation models are trained purely on unstructured 2D data, limiting their understanding of 3D structure of objects and scenes. In this work, we show that fine-tuning on 3D-aware data improves the quality of emerging semantic features. We design a method to lift semantic 2D features into an efficient 3D Gaussian representation, which allows us to re-render them for arbitrary views. Using the rendered 3D-aware features, we design a fine-tuning strategy to transfer such 3D awareness into a 2D foundation model. We demonstrate that models fine-tuned in that way produce features that readily improve downstream task performance in semantic segmentation and depth estimation through simple linear probing. Notably, though fined-tuned on a single indoor dataset, the improvement is transferable to a variety of indoor datasets and out-of-domain datasets. We hope our study encourages the community to consider injecting 3D awareness when training 2D foundation models. Project page: https://ywyue.github.io/FiT3D.<|reference_end|>
@article{yue2024improving,
    title={Improving 2D Feature Representations by 3D-Aware Fine-Tuning},
    author={Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, Jan Eric
  Lenssen},
    journal={arXiv preprint arXiv:2407.20229},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.20229},
    primaryClass={cs.CV}
}