Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation
<|reference_start|>Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation: Open-vocabulary semantic segmentation aims at segmenting arbitrary categories expressed in textual form. Previous works have trained over large amounts of image-caption pairs to enforce pixel-level multimodal alignments. However, captions provide global information about the semantics of a given image but lack direct localization of individual concepts. Further, training on large-scale datasets inevitably brings significant computational costs. In this paper, we propose FreeDA, a training-free diffusion-augmented method for open-vocabulary semantic segmentation, which leverages the ability of diffusion models to visually localize generated concepts and local-global similarities to match class-agnostic regions with semantic classes. Our approach involves an offline stage in which textual-visual reference embeddings are collected, starting from a large set of captions and leveraging visual and semantic contexts. At test time, these are queried to support the visual matching process, which is carried out by jointly considering class-agnostic regions and global semantic similarities. Extensive analyses demonstrate that FreeDA achieves state-of-the-art performance on five datasets, surpassing previous methods by more than 7.0 average points in terms of mIoU and without requiring any training.<|reference_end|>
@article{barsellotti2024training-free,
    title={Training-Free Open-Vocabulary Segmentation with Offline
  Diffusion-Augmented Prototype Generation},
    author={Luca Barsellotti, Roberto Amoroso, Marcella Cornia, Lorenzo Baraldi,
  Rita Cucchiara},
    journal={arXiv preprint arXiv:2404.06542},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.06542},
    primaryClass={cs.CV}
}