LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision
  Foundation Models
Vision foundation models (VFMs) such as DINOv2 and CLIP have achieved
impressive results on various downstream tasks, but their limited feature
resolution hampers performance in applications requiring pixel-level
understanding. Feature upsampling offers a promising direction to address this
challenge. In this work, we identify two critical factors for enhancing feature
upsampling: the upsampler architecture and the training objective. For the
upsampler architecture, we introduce a coordinate-based cross-attention
transformer that integrates the high-resolution images with coordinates and
low-resolution VFM features to generate sharp, high-quality features. For the
training objective, we propose constructing high-resolution pseudo-groundtruth
features by leveraging class-agnostic masks and self-distillation. Our approach
effectively captures fine-grained details and adapts flexibly to various input
and feature resolutions. Through experiments, we demonstrate that our approach
significantly outperforms existing feature upsampling techniques across various
downstream tasks. Our code is released at https://github.com/andrehuang/loftup.
6807f86ad9efafd30d1ae9ff
Haiwen Huang
6807f86ad9efafd30d1aea00
Anpei Chen
6807f86ad9efafd30d1aea01
Volodymyr Havrylov
6807f86ad9efafd30d1aea02
Andreas Geiger
6807f86ad9efafd30d1aea03
Dan Zhang
LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision
  Foundation Models
Vision foundation models (VFMs) such as DINOv2 and CLIP have achieved
impressive results on various downstream tasks, but their limited feature
resolution hampers performance in applications requiring pixel-level
understanding. Feature upsampling offers a promising direction to address this
challenge. In this work, we identify two critical factors for enhancing feature
upsampling: the upsampler architecture and the training objective. For the
upsampler architecture, we introduce a coordinate-based cross-attention
transformer that integrates the high-resolution images with coordinates and
low-resolution VFM features to generate sharp, high-quality features. For the
training objective, we propose constructing high-resolution pseudo-groundtruth
features by leveraging class-agnostic masks and self-distillation. Our approach
effectively captures fine-grained details and adapts flexibly to various input
and feature resolutions. Through experiments, we demonstrate that our approach
significantly outperforms existing feature upsampling techniques across various
downstream tasks. Our code is released at https://github.com/andrehuang/loftup.
A coordinate-based cross-attention transformer and a class-agnostic masking with self-distillation approach improve feature upsampling in vision foundation models, enhancing performance in pixel-level understanding tasks.
vision foundation models DINOv2 CLIP feature upsampling coordinate-based cross-attention transformer high-resolution pseudo-groundtruth features class-agnostic masks self-distillation