Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox
<|reference_start|>Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox: Most existing out-of-distribution (OOD) detection benchmarks classify samples with novel labels as the OOD data. However, some marginal OOD samples actually have close semantic contents to the in-distribution (ID) sample, which makes determining the OOD sample a Sorites Paradox. In this paper, we construct a benchmark named Incremental Shift OOD (IS-OOD) to address the issue, in which we divide the test samples into subsets with different semantic and covariate shift degrees relative to the ID dataset. The data division is achieved through a shift measuring method based on our proposed Language Aligned Image feature Decomposition (LAID). Moreover, we construct a Synthetic Incremental Shift (Syn-IS) dataset that contains high-quality generated images with more diverse covariate contents to complement the IS-OOD benchmark. We evaluate current OOD detection methods on our benchmark and find several important insights: (1) The performance of most OOD detection methods significantly improves as the semantic shift increases; (2) Some methods like GradNorm may have different OOD detection mechanisms as they rely less on semantic shifts to make decisions; (3) Excessive covariate shifts in the image are also likely to be considered as OOD for some methods. Our code and data are released in https://github.com/qqwsad5/IS-OOD.<|reference_end|>
@article{long2024rethinking,
    title={Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites
  Paradox},
    author={Xingming Long, Jie Zhang, Shiguang Shan, Xilin Chen},
    journal={arXiv preprint arXiv:2406.09867},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.09867},
    primaryClass={cs.CV}
}