This work presents a method for active anomaly detection which can be built upon existing deep learning solutions for unsupervised anomaly detection. We show that a prior needs to be assumed on what the anomalies are, in order to have performance guarantees in unsupervised anomaly detection. We argue that active anomaly detection has, in practice, the same cost of unsupervised anomaly detection but with the possibility of much better results. To solve this problem, we present a new layer that can be attached to any deep learning model designed for unsupervised anomaly detection to transform it into an active method, presenting results on both synthetic and real anomaly detection datasets.
(Since the reviewer was unclear about the OpenReview process, this review was earlier posted as public comment)
Most claims of novelty can be clearly refuted such as the first sentence of the abstract "...This work presents a new approach to active anomaly detection..." and the paper does not give due credit to existing work. Current research such as Das et al. which is the most relevant has been deliberately not introduced upfront with other works (because it shows lack of the present paper's novelty) and instead deferred to later sections. The onus of a thorough literature review and laying down a proper context is on the authors, not the reviewers. Detailed comments are below.
      
      1. Related Works: "...active anomaly detection remains an under-explored approach to this problem..."
          - Active learning in anomaly detection is well-researched (AI2, etc.). See related works section in Das et al. 2016 and:
            - K. Veeramachaneni, I. Arnaldo, A. Cuesta-Infante, V. Korrapati, C. Bassias, and K. Li, "Ai2: Training a big data machine to defend," International Conference on Big Data Security, 2016.
        
      2. "To deal with the cold start problem, for the first 10 calls of select_top...":
          - No principled approach to deal with cold start and one-sided labels (i.e., the ability to use labels when instances from only one class are labeled.)
        
      3. Many arbitrary hyper parameters as compared to simpler techniques:
          - The number of layers, nodes in hidden layers.
            - The number of instances (k) per iteration
            - The number of pretraining iterations
            - The number of times the network is retrained (100) after each labeling call
            - Dealing with cold start (10 labeling iterations of 10 labels each, i.e. 100 labels).
        
      4. The paper mentions that s(x) might not be differentiable. However, the sigmoid form of s(x) is differentiable.
      
      5. Does not acknowledge the well-known result that mixture models are unidentifiable. The math in the paper is mostly redundant. Some references:
          - Identifiability  Of  Nonparametric  Mixture  Models And  Bayes  Optimal  Clustering (pradeepr/arxiv npmix v.pdf)" target="_blank" rel="nofollow">https://www.cs.cmu.edu/ pradeepr/arxiv npmix v.pdf)
          - Semiparametric estimation of a two-component mixture model by Bordes, L., Kojadinovic, I., and Vandekerkhove, P., Annals of Statistics, 2006 (https://arxiv.org/pdf/math/0607812.pdf)
          - Inference for mixtures of symmetric distributions by David R. Hunter, Shaoli Wang, Thomas P. Hettmansperger, Annals of Statistics, 2007 (https://arxiv.org/pdf/0708.0499.pdf)
          - Inference on Mixtures Under Tail Restrictions by K. Jochmans, M. Henry, and B. Salanie, Econometric Theory, 2017 (http://econ.sciences-po.fr/sites/default/files/file/Inference.pdf)
          
      6. Does not acknowledge existing work that adds classifier over unsupervised detectors (such as AI2). This is very common.
        - This is another linear model (logistic) on top of transformed features. The difference is that the transformed features are from a neural network and optimization can be performed in a joint fashion. The novelty is marginal.
        
      7. While the paper argues that a prior needs to be assumed, it does not use any in the algorithm. There seems to be a disconnect. It also does not acknowledge that AAD (LODA/Tree) does use a prior. Priors for anomaly proportions in unsupervised algorithms are well-known (most AD algos support that such as OC-SVM, Isolation Forest, LOF, etc.).
        
      8. Does not compare against current state-of-the-art Tree-based AAD
          - Incorporating Expert Feedback into Tree-based Anomaly Detection by Das et al., KDD, 2017.
        
      9. The 'Generalized' in the title is incorrect and misleading. This is specific to deep-networks. Stacking supervised classifiers on unsupervised detectors is very common. See comments on related works.
      
      10. Does not propose any other query strategies than greedily selecting top.
      
      11. Question: Does this support streaming?