LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary
  Semantic Segmentation
We propose a training-free method for open-vocabulary semantic segmentation
using Vision-and-Language Models (VLMs). Our approach enhances the initial
per-patch predictions of VLMs through label propagation, which jointly
optimizes predictions by incorporating patch-to-patch relationships. Since VLMs
are primarily optimized for cross-modal alignment and not for intra-modal
similarity, we use a Vision Model (VM) that is observed to better capture these
relationships. We address resolution limitations inherent to patch-based
encoders by applying label propagation at the pixel level as a refinement step,
significantly improving segmentation accuracy near class boundaries. Our
method, called LPOSS+, performs inference over the entire image, avoiding
window-based processing and thereby capturing contextual interactions across
the full image. LPOSS+ achieves state-of-the-art performance among
training-free methods, across a diverse set of datasets. Code:
https://github.com/vladan-stojnic/LPOSS
67e3d807304f166b6660db63
Vladan Stojnić
67e3d807304f166b6660db64
Yannis Kalantidis
67e3d807304f166b6660db65
Jiří Matas
67e3d807304f166b6660db66
Giorgos Tolias
LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary
  Semantic Segmentation
We propose a training-free method for open-vocabulary semantic segmentation
using Vision-and-Language Models (VLMs). Our approach enhances the initial
per-patch predictions of VLMs through label propagation, which jointly
optimizes predictions by incorporating patch-to-patch relationships. Since VLMs
are primarily optimized for cross-modal alignment and not for intra-modal
similarity, we use a Vision Model (VM) that is observed to better capture these
relationships. We address resolution limitations inherent to patch-based
encoders by applying label propagation at the pixel level as a refinement step,
significantly improving segmentation accuracy near class boundaries. Our
method, called LPOSS+, performs inference over the entire image, avoiding
window-based processing and thereby capturing contextual interactions across
the full image. LPOSS+ achieves state-of-the-art performance among
training-free methods, across a diverse set of datasets. Code:
https://github.com/vladan-stojnic/LPOSS
LPOSS+ is a training-free semantic segmentation method that enhances Vision-and-Language Models with label propagation at the pixel level, achieving state-of-the-art results.
Vision-and-Language Models label propagation per-patch predictions patch-to-patch relationships Vision Model intra-modal similarity pixel-level refinement cross-modal alignment contextual interactions