title: LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision
    Foundation Models
summary: Vision foundation models (VFMs) such as DINOv2 and CLIP have achieved
  impressive results on various downstream tasks, but their limited feature
  resolution hampers performance in applications requiring pixel-level
  understanding. Feature upsampling offers a promising direction to address this
  challenge. In this work, we identify two critical factors for enhancing feature
  upsampling: the upsampler architecture and the training objective. For the
  upsampler architecture, we introduce a coordinate-based cross-attention
  transformer that integrates the high-resolution images with coordinates and
  low-resolution VFM features to generate sharp, high-quality features. For the
  training objective, we propose constructing high-resolution pseudo-groundtruth
  features by leveraging class-agnostic masks and self-distillation. Our approach
  effectively captures fine-grained details and adapts flexibly to various input
  and feature resolutions. Through experiments, we demonstrate that our approach
  significantly outperforms existing feature upsampling techniques across various
  downstream tasks. Our code is released at https://github.com/andrehuang/loftup.
paper_authors._id: 6807f86ad9efafd30d1ae9ff
paper_authors.name: Haiwen Huang
paper_authors._id: 6807f86ad9efafd30d1aea00
paper_authors.name: Anpei Chen
paper_authors._id: 6807f86ad9efafd30d1aea01
paper_authors.name: Volodymyr Havrylov
paper_authors._id: 6807f86ad9efafd30d1aea02
paper_authors.name: Andreas Geiger
paper_authors._id: 6807f86ad9efafd30d1aea03
paper_authors.name: Dan Zhang
paper_title: LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision
    Foundation Models
paper_summary: Vision foundation models (VFMs) such as DINOv2 and CLIP have achieved
  impressive results on various downstream tasks, but their limited feature
  resolution hampers performance in applications requiring pixel-level
  understanding. Feature upsampling offers a promising direction to address this
  challenge. In this work, we identify two critical factors for enhancing feature
  upsampling: the upsampler architecture and the training objective. For the
  upsampler architecture, we introduce a coordinate-based cross-attention
  transformer that integrates the high-resolution images with coordinates and
  low-resolution VFM features to generate sharp, high-quality features. For the
  training objective, we propose constructing high-resolution pseudo-groundtruth
  features by leveraging class-agnostic masks and self-distillation. Our approach
  effectively captures fine-grained details and adapts flexibly to various input
  and feature resolutions. Through experiments, we demonstrate that our approach
  significantly outperforms existing feature upsampling techniques across various
  downstream tasks. Our code is released at https://github.com/andrehuang/loftup.
paper_ai_summary: A coordinate-based cross-attention transformer and a class-agnostic masking with self-distillation approach improve feature upsampling in vision foundation models, enhancing performance in pixel-level understanding tasks.
paper_ai_keywords: vision foundation models DINOv2 CLIP feature upsampling coordinate-based cross-attention transformer high-resolution pseudo-groundtruth features class-agnostic masks self-distillation