Deep Anomaly Detection Using Geometric Transformations
We consider the problem of anomaly detection in images, and present a new detection technique. Given a sample of images, all known to belong to a "normal" class (e.g., dogs), we show how to train a deep neural model that can detect out-of-distribution images (i.e., non-dog objects). The main idea behind our scheme is to train a multi-class model to discriminate between dozens of geometric transformations applied on all the given images. The auxiliary expertise learned by the model generates feature detectors that effectively identify, at test time, anomalous images based on the softmax activation statistics of the model when applied on transformed images. We present extensive experiments using the proposed detector, which indicate that our technique consistently improves all known algorithms by a wide margin.
The authors proposed an original approach to image anomaly detection. They proposed to define a set of geometric transformations and then to construct a classifier to predict to which type of transformation a given image is more similar. In case the classifier score is low, the image is considered as anomalous.   As a test set the authors considered standard image datasets like CIFAR, Fashion-MNIST and CatsVsDogs. However, there exist special datasets to evaluate anomaly detection methods for images. E.g.   http://odds.cs.stonybrook.edu/#table5  The UCSD anomaly detection is annotated: each video frame is annotated as anomalous/non-anomalous and sometimes it contains pixel-wise annotations. Such datasets are closer to real-life applications and should be used for testing.  As baselines the authors used  - one-class SVM based either on pixels, or on low-dimension representation from the neural network - energy-based models - deep autoencoding GMM - GANs The authors claimed that in case of one-class SVM they used the best hyperparameters as there is no approach how to tune them. In fact, this is not true. There are approaches how to tune hyperparameters of one-class SVM: - Wang, S., Liu, Q., et al.: Hyperparameter selection of one-class support vector machine by self-adaptive data shifting. Pattern Recognition 74 (2018) 198–211 - Thomas, A., Feuillard, V., et al.: Calibration of one-class svm for mv set estimation.  CoRR abs/1508.07535 (2015)  - E. Burnaev, P. Erofeev, D. Smolyakov. Model Selection for Anomaly Detection // Proc. SPIE 9875, Eighth International Conference on Machine Vision, 987525 (December 8, 2015); 5 P. doi:10.1117/12.2228794; http://dx.doi.org/10.1117/12.2228794  Moreover, additional gain in accuracy when using one-class SVM can be obtained through end-to-end learning of embeddings specifically for one-class classification. There exists an approach how to do that, namely,  https://arxiv.org/pdf/1804.04888.pdf Also see https://arxiv.org/pdf/1801.05365.pdf  Of course, the one-class SVM is not the most powerful approach for anomaly detection. Experiments on numerous benchmark datasets with ground truth that compared popular anomaly detection algorithms find that One Class SVM ranks at the bottom, see  We note that the top performer in [1] is the Isolation Forest algorithm [2], an ensemble of randomized trees. Thus it is important to include in the comparison Isolation Forest algorithm based on the same features, as one-class SVM!  [1] Emmott, A., Das, S., et al.: Systematic construction of anomaly detection bench-marks from real data. In: Proceedings of KDD ODD. (2013) [2] Liu, F., Ting, K., Zhou, Z.H.: Isolation forest. In: Proceedings of ICDM. (2008)  - In general, features, generated by neural networks, are believed to provide efficient characterization of image data. In particular, very often it is enough to use linear classifier in the feature space, produce by deep layers of the neural network. Thus, why did the authors use kernel one-class SVM (CAE-OC-SVM)? It seems that it can be enough to use linear one-class SVM with features, produced by usual deep network such as VGG.  - another issue concerning section 5.1. PCA should be used as a baseline! It is very often used in image processing engineering applications, including anomaly detection.  - page 3, line 104. The authors proposed to use AUC. However, this may be not appropriate measure to estimate performance of anomaly detector. In fact, in real datasets the number of anomalies is small compareв to the size of the normal class. Thus area under the precision-recall curve is better suited for such imbalanced cases.  - displayed formula after the line 100: after the first line of the displayed formula a comma sign should be used  - The idea to use some class of geometric transformations T looks similar to the idea of image set augmentation. It is obvious that “richness” of the used class of geometric transformations should significantly influence anomaly detection accuracy. However, the authors did not investigate how anomaly detection accuracy depend on the diversity of the used class of geometric transformations. To which extent obtained anomaly detection results are sensitive to properties of this class?  - Practical image classification problems often contain a lot of classes (so-called extreme classification problems). Is the proposed method capable to deal with such case? To which extent obtained anomaly detection performance is sensitive to the number of classes, used to model anomalous observations?  - in section 5.3 the authors consider issues about hyperparameters. Are there any recommendations how to select a particular architecture of the neural network? Should we try to find such architecture which provides as accurate classification of types of image transformations as possible?   - in the paper by Liang et al. “Enhancing the reliability …” they proposed the method that using temperature scaling and adding small perturbations to the input can separate the softmax score distributions between in- and out-of-distribution images, allowing for effective detection of out-of-distribution images. In fact, in the current paper the authors proposed to use a set of geometric transformation as such perturbations.   Conclusion: - the topic of the paper is important - the paper is well-written. However, still there are a number of open questions, see comments above. - the results are novel. However, the novelty is not very big compared, e.g. to the results of the paper by Liang et al.