title: LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary
    Semantic Segmentation
summary: We propose a training-free method for open-vocabulary semantic segmentation
  using Vision-and-Language Models (VLMs). Our approach enhances the initial
  per-patch predictions of VLMs through label propagation, which jointly
  optimizes predictions by incorporating patch-to-patch relationships. Since VLMs
  are primarily optimized for cross-modal alignment and not for intra-modal
  similarity, we use a Vision Model (VM) that is observed to better capture these
  relationships. We address resolution limitations inherent to patch-based
  encoders by applying label propagation at the pixel level as a refinement step,
  significantly improving segmentation accuracy near class boundaries. Our
  method, called LPOSS+, performs inference over the entire image, avoiding
  window-based processing and thereby capturing contextual interactions across
  the full image. LPOSS+ achieves state-of-the-art performance among
  training-free methods, across a diverse set of datasets. Code:
  https://github.com/vladan-stojnic/LPOSS
paper_authors._id: 67e3d807304f166b6660db63
paper_authors.name: Vladan Stojnić
paper_authors._id: 67e3d807304f166b6660db64
paper_authors.name: Yannis Kalantidis
paper_authors._id: 67e3d807304f166b6660db65
paper_authors.name: Jiří Matas
paper_authors._id: 67e3d807304f166b6660db66
paper_authors.name: Giorgos Tolias
paper_title: LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary
    Semantic Segmentation
paper_summary: We propose a training-free method for open-vocabulary semantic segmentation
  using Vision-and-Language Models (VLMs). Our approach enhances the initial
  per-patch predictions of VLMs through label propagation, which jointly
  optimizes predictions by incorporating patch-to-patch relationships. Since VLMs
  are primarily optimized for cross-modal alignment and not for intra-modal
  similarity, we use a Vision Model (VM) that is observed to better capture these
  relationships. We address resolution limitations inherent to patch-based
  encoders by applying label propagation at the pixel level as a refinement step,
  significantly improving segmentation accuracy near class boundaries. Our
  method, called LPOSS+, performs inference over the entire image, avoiding
  window-based processing and thereby capturing contextual interactions across
  the full image. LPOSS+ achieves state-of-the-art performance among
  training-free methods, across a diverse set of datasets. Code:
  https://github.com/vladan-stojnic/LPOSS
paper_ai_summary: LPOSS+ is a training-free semantic segmentation method that enhances Vision-and-Language Models with label propagation at the pixel level, achieving state-of-the-art results.
paper_ai_keywords: Vision-and-Language Models label propagation per-patch predictions patch-to-patch relationships Vision Model intra-modal similarity pixel-level refinement cross-modal alignment contextual interactions