title: Improving 2D Feature Representations by 3D-Aware Fine-Tuning
summary: Current visual foundation models are trained purely on unstructured 2D data,
  limiting their understanding of 3D structure of objects and scenes. In this
  work, we show that fine-tuning on 3D-aware data improves the quality of
  emerging semantic features. We design a method to lift semantic 2D features
  into an efficient 3D Gaussian representation, which allows us to re-render them
  for arbitrary views. Using the rendered 3D-aware features, we design a
  fine-tuning strategy to transfer such 3D awareness into a 2D foundation model.
  We demonstrate that models fine-tuned in that way produce features that readily
  improve downstream task performance in semantic segmentation and depth
  estimation through simple linear probing. Notably, though fined-tuned on a
  single indoor dataset, the improvement is transferable to a variety of indoor
  datasets and out-of-domain datasets. We hope our study encourages the community
  to consider injecting 3D awareness when training 2D foundation models. Project
  page: https://ywyue.github.io/FiT3D.
paper_authors._id: 66a854d99189298acbf49941
paper_authors.name: Yuanwen Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T07:25:49.460Z
paper_authors._id: 65b0b33a403a23a2fda96cbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65b0b33a403a23a2fda96cbc/f-45UN20BixueeJPAuJUj.png
paper_authors.fullname: Yuanwen Yue
paper_authors.type: user
paper_authors.user: yuanwenyue
paper_authors._id: 66a854d99189298acbf49942
paper_authors.name: Anurag Das
paper_authors._id: 66a854d99189298acbf49943
paper_authors.name: Francis Engelmann
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-18T08:48:31.971Z
paper_authors._id: 671a7160e45d92e304e82f45
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-trHIg97qX0dE9T4UYycc.png
paper_authors.fullname: Francis Engelmann
paper_authors.type: user
paper_authors.user: francisengelmann
paper_authors._id: 66a854d99189298acbf49944
paper_authors.name: Siyu Tang
paper_authors._id: 66a854d99189298acbf49945
paper_authors.name: Jan Eric Lenssen
paper_title: Improving 2D Feature Representations by 3D-Aware Fine-Tuning
paper_summary: Current visual foundation models are trained purely on unstructured 2D data,
  limiting their understanding of 3D structure of objects and scenes. In this
  work, we show that fine-tuning on 3D-aware data improves the quality of
  emerging semantic features. We design a method to lift semantic 2D features
  into an efficient 3D Gaussian representation, which allows us to re-render them
  for arbitrary views. Using the rendered 3D-aware features, we design a
  fine-tuning strategy to transfer such 3D awareness into a 2D foundation model.
  We demonstrate that models fine-tuned in that way produce features that readily
  improve downstream task performance in semantic segmentation and depth
  estimation through simple linear probing. Notably, though fined-tuned on a
  single indoor dataset, the improvement is transferable to a variety of indoor
  datasets and out-of-domain datasets. We hope our study encourages the community
  to consider injecting 3D awareness when training 2D foundation models. Project
  page: https://ywyue.github.io/FiT3D.
paper_ai_summary: Injecting 3D awareness by lifting 2D features into a 3D Gaussian representation enhances semantic understanding and downstream task performance in 2D foundation models.
paper_ai_keywords: Gaussian representation 3D Gaussian representation semantic segmentation depth estimation linear probing