Despite having excellent performances for a wide variety of tasks, modern neural networks are unable to provide a prediction with a reliable confidence estimate which would allow to detect misclassifications. This limitation is at the heart of what is known as an adversarial example, where the network provides a wrong prediction associated with a strong confidence to a slightly modified image. Moreover, this overconfidence issue has also been observed for out-of-distribution data. We show through several experiments that the softmax activation, usually placed as the last layer of modern neural networks, is partly responsible for this behaviour. We give qualitative insights about its impact on the MNIST dataset, showing that relevant information present in the logits is lost once the softmax function is applied. The same observation is made through quantitative analysis, as we show that two out-of-distribution and adversarial example detectors obtain competitive results when using logit values as inputs, but provide considerably lower performances if they use softmax probabilities instead: from 98.0% average AUROC to 56.8% in some settings. These results provide evidence that the softmax activation hinders the detection of adversarial and out-of-distribution examples, as it masks a significant part of the relevant information present in the logits.
This paper suggests that the logits cary more information than the maximum softmax probability for OOD detection. They suggest this with scatterplots and develop techniques to support this claim.
They use logits for as features for OOD detection with by using a kernel density estimator. They also use a NN trained with logits features, but this assumes we can peak at the test distribution, so I ignore this entirely in this evaluation.
Unfortunately their KDE density estimator does not perform better than the maximum softmax probability for OOD detection on CIFAR-10 (74.3 vs 91.7).
They do not show results on CIFAR-100, but since the dimensionality of the logits would increase by an order of magnitude, one would expect kernel density estimation to perform much worse. The authors should include an evaluation on CIFAR-100 for completeness.
Small comments:
Table 3 is a comment about featurization. Does this hold when taking the log of the softmax probabilities (not the same as logits)? If not, then this isn't much a count against the softmax per se. Even then, this is a comment on using softmax information for KDE, not using the maximum softmax probability itself for OOD detection.
The full results for Table 2 are needed. Perhaps place this in an appendix.
They repeat that the logits contain more information than the maximum softmax probability, but so does the raw input. The challenge is not introducing more noise/variance when introducing more information.
I was confused at the experimental description. The information should be contained in one location. They train one of their CIFAR networks for 30 epochs, which isn't enough training time. Consequently, I suspect that those results are not worth drawing implications from since the accuracy is presumably low.