Generative Probabilistic Novelty Detection with Adversarial Autoencoders
Novelty detection is the problem of identifying whether a new data point is considered to be an inlier or an outlier. We assume that training data is available to describe only the inlier distribution. Recent approaches primarily leverage deep encoder-decoder network architectures to compute a reconstruction error that is used to either compute a novelty score or to train a one-class classifier. While we too leverage a novel network of that kind, we take a probabilistic approach and effectively compute how likely it is that a sample was generated by the inlier distribution. We achieve this with two main contributions. First, we make the computation of the novelty probability feasible because we linearize the parameterized manifold capturing the underlying structure of the inlier distribution, and show how the probability factorizes and can be computed with respect to local coordinates of the manifold tangent space. Second, we improve the training of the autoencoder network. An extensive set of results show that the approach achieves state-of-the-art performance on several benchmark datasets.
The problem of the study is outlier detection given only inlier training data. The high-level approach is to learn a density estimation function over the training data, and then filter out outliers using a learned threshold \gamma. They approximate the density function through a decomposition over the tangent space of the learned manifold near each given sample. To learn the manifold structure the authors use a variation of adversarial autoencoders. The evaluation is performed on MNIST, FashionMNIST, and COIL against a few baselines.  Overall, the paper is very well-written and easy to follow -- the presentation progresses coherently. However, there are several issues that I think must be addressed before the work is published.  At the heart of this work, the problem being addressed is that of density estimation. The learned function is then used to perform outlier detection. - There are several existing methods that are capable of performing density estimation. Autoregressive models such as PixelCNN, PixelCNN++, and the previous work, most of the recent compression techniques that use deep neural networks, and even a simple Variational Autoencoder can give us density estimates. There should be at least one of these density estimation methods to assess how well the proposed method approximates the density to determine whether the trouble of implementing such a complex technique is worth it. - The outlier detection problem is a quickly growing area which is not easy to track, but the authors should compare against easy-to-implement state-of-the-art methods such as ODIN [1] (ICLR 2018) at least. Thresholding the prediction of a predictive classifier, thresholding the reconstruction error of an AE that is trained with the same pipeline (without the additional loss functions) are two other easy to implement methods that they could compare against. - The presented evaluation is too weak. There are several measures in the outlier detection literature, from which the authors only report one or two. They don't adopt the same evaluation protocol either, yet, they use the numbers from other evaluations as a comparison. Furthermore, the evaluation is only performed on low-dimensional, low-complexity datasets MNIST, FashionMNIST, and COIL. There should be at least a CIFAR10 in the evaluation. It is common in the literature to also compare against CIFAR100 and downscaled imagenet.  Quality. The submission is technically sound. The idea is interesting and the work is executed and presented well and coherently. However, as mentioned earlier, at a high-level, the direction of the work should be slightly corrected to better position it within the existing literature.  Clarity. The paper is well-written and clear. The previous work could be significantly improved. In specific, I recommend adding the density estimation literature, and out-of-distribution detection literature as the method is addressing both of the problems. The mentioned work in the review could serve as a good starting point in these areas. The evaluation could also be improved as I discussed earlier.  Originality. I haven't seen this idea executed before. It is both novel and interesting. It would be influential if it improves the density approximation problem or performs outlier detection better (shown more strongly in evaluation). Even if it does not outperform the other methods, there still needs to be a visible connection and adequate comparison to position the work properly in the literature.  Significance. Unfortunately, I think, the work in its current state is unlikely to have an impact on the relevant areas.  References. [1] Liang, Shiyu, Yixuan Li, and R. Srikant. "Enhancing the reliability of out-of-distribution image detection in neural networks." arXiv preprint arXiv:1706.02690 (2017). -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Update after rebuttal. I think the author response has adequately addressed the raised concerns -- the paper would be a good submission with the proposed revisions. I have increased the overall score to 7.