Versatile Anomaly Detection with Outlier Preserving Distribution Mapping Autoencoders
State-of-the-art deep learning methods for outlier detection make the assumption that anomalies will appear far away from inlier data in the latent space produced by distribution mapping deep networks. However, this assumption fails in practice, because the divergence penalty adopted for this purpose encourages mapping outliers into the same high-probability regions as inliers. To overcome this shortcoming, we introduce a novel deep learning outlier detection method, called Outlier Preserving Distribution Mapping Autoencoder (OP-DMA), which succeeds to map outliers to low probability regions in the latent space of an autoencoder. For this we leverage the insight that outliers are likely to have a higher reconstruction error than inliers. We thus achieve outlier-preserving distribution mapping through weighting the reconstruction error of individual points by the value of a multivariate Gaussian probability density function evaluated at those points. This weighting implies that outliers will result overall penalty if they are mapped to low-probability regions. We show that if the global minimum of our newly proposed loss function is achieved, then our OP-DMA maps inliers to regions with a Mahalanobis distance less than delta, and outliers to regions past this delta, delta being the inverse Chi Squared CDF evaluated at (1-alpha) with alpha the percentage of outliers in the dataset. Our experiments confirm that OP-DMA consistently outperforms the state-of-art methods on a rich variety of outlier detection benchmark datasets.
The paper proposes an improved extension of the Wasserstein auto-encoder for anomaly detection. The novelty is in proposing a weighted reconstruction error that penalizes the mapping of data with high reconstruction errors (mostly anomalies) into high probability regions. The idea being that an outlier would have a higher reconstruction error, and hence should be mapped to low-probability region of the latent distribution. Experimental Results: As a distribution mapping auto-encoder model, OP-DMA outperforms the deep learning based state-of-the-art models in the same domain. Overall Assessment: The authors have a nice idea of forcing the latent mappings of inputs to correlate with their reconstruction error. Overall, the method is promising, but I have the following concerns: * Using the reconstruction error as an anomaly score has been explored many years ago (check replicator neural networks), the novelty here is to enforce that on the latent space in the context of a variational auto-encoder. I am not sure if, from anomaly detection perspective, this is any better than simply using the reconstruction score. Why go the VAE route at all? * Is there a possibility that assuming a single multi-variate Gaussian, as a prior, too restrictive? Could it result in a high false alarm rate as well? I guess this could be answered by more experimental results on richer data sets (even synthetic is fine). * In most score based algorithms, the anomaly score is computed without assuming any prior knowledge about the contamination proportion. However, in the case of OP-DMA, the contamination parameter is used to train the auto-encoder that scores the data. This might result in an optimization that is very specific to the parameter setting. I strongly recommend a sensitivity analysis to study the robustness of the model against different values of contamination parameter. * Performance on synthetic data-set has not been presented. The set H in theorem 3 has not been defined. * Additional comparison with other non-distribution mapping state-of-the-art models such as LOF, oc-SVM, KNN would give a clearer idea of the performance. This is important, because in my past experience, non-deep learning methods give much better results on the benchmark data sets that the authors have evaluated their method on. In fact, a comparative analysis (See - https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152173) gives a very nice comparison. However, since the authors provide results using Avg F1-score, instead of AUC curve, it was not possible to compare them myself. * In figure 3, in the training process, the authors have describe to add the divergence between the latent and prior distribution to the loss function, however, nothing like this is clearly shown in the figure.The references of figures in the text are either out of place or incorrect. Figure 1(a) and (b) in reference to the text are incorrect. Figure 2 is the misleading figure as it doesn*t illustrate the anomaly detection process. Figure 3 has not been mentioned anywhere in the text. The authors have mentioned the comparison of their method with Wasserstien and variational auto-encoders in the text, while in table 2 and 4, AAE has also been shown as one of the method for comparison, which is never mentioned or described in the text. * Typo in caption of Figure 1 and the first line of section 3.3 Overall, I am hesitant to recommend the paper before cross-checking the issue with contamination proportion and learning more about how a VAE framework is indeed important for anomaly detection.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Versatile Anomaly Detection with Outlier Preserving Distribution Mapping Autoencoders
State-of-the-art deep learning methods for outlier detection make the assumption that anomalies will appear far away from inlier data in the latent space produced by distribution mapping deep networks. However, this assumption fails in practice, because the divergence penalty adopted for this purpose encourages mapping outliers into the same high-probability regions as inliers. To overcome this shortcoming, we introduce a novel deep learning outlier detection method, called Outlier Preserving Distribution Mapping Autoencoder (OP-DMA), which succeeds to map outliers to low probability regions in the latent space of an autoencoder. For this we leverage the insight that outliers are likely to have a higher reconstruction error than inliers. We thus achieve outlier-preserving distribution mapping through weighting the reconstruction error of individual points by the value of a multivariate Gaussian probability density function evaluated at those points. This weighting implies that outliers will result overall penalty if they are mapped to low-probability regions. We show that if the global minimum of our newly proposed loss function is achieved, then our OP-DMA maps inliers to regions with a Mahalanobis distance less than delta, and outliers to regions past this delta, delta being the inverse Chi Squared CDF evaluated at (1-alpha) with alpha the percentage of outliers in the dataset. Our experiments confirm that OP-DMA consistently outperforms the state-of-art methods on a rich variety of outlier detection benchmark datasets.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper proposes an improved extension of the Wasserstein auto-encoder for anomaly detection. The novelty is in proposing a weighted reconstruction error that penalizes the mapping of data with high reconstruction errors (mostly anomalies) into high probability regions. The idea being that an outlier would have a higher reconstruction error, and hence should be mapped to low-probability region of the latent distribution. Experimental Results: As a distribution mapping auto-encoder model, OP-DMA outperforms the deep learning based state-of-the-art models in the same domain. Overall Assessment: The authors have a nice idea of forcing the latent mappings of inputs to correlate with their reconstruction error. Overall, the method is promising, but I have the following concerns: * Using the reconstruction error as an anomaly score has been explored many years ago (check replicator neural networks), the novelty here is to enforce that on the latent space in the context of a variational auto-encoder. I am not sure if, from anomaly detection perspective, this is any better than simply using the reconstruction score. Why go the VAE route at all? * Is there a possibility that assuming a single multi-variate Gaussian, as a prior, too restrictive? Could it result in a high false alarm rate as well? I guess this could be answered by more experimental results on richer data sets (even synthetic is fine). * In most score based algorithms, the anomaly score is computed without assuming any prior knowledge about the contamination proportion. However, in the case of OP-DMA, the contamination parameter is used to train the auto-encoder that scores the data. This might result in an optimization that is very specific to the parameter setting. I strongly recommend a sensitivity analysis to study the robustness of the model against different values of contamination parameter. * Performance on synthetic data-set has not been presented. The set H in theorem 3 has not been defined. * Additional comparison with other non-distribution mapping state-of-the-art models such as LOF, oc-SVM, KNN would give a clearer idea of the performance. This is important, because in my past experience, non-deep learning methods give much better results on the benchmark data sets that the authors have evaluated their method on. In fact, a comparative analysis (See - https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152173) gives a very nice comparison. However, since the authors provide results using Avg F1-score, instead of AUC curve, it was not possible to compare them myself. * In figure 3, in the training process, the authors have describe to add the divergence between the latent and prior distribution to the loss function, however, nothing like this is clearly shown in the figure.The references of figures in the text are either out of place or incorrect. Figure 1(a) and (b) in reference to the text are incorrect. Figure 2 is the misleading figure as it doesn*t illustrate the anomaly detection process. Figure 3 has not been mentioned anywhere in the text. The authors have mentioned the comparison of their method with Wasserstien and variational auto-encoders in the text, while in table 2 and 4, AAE has also been shown as one of the method for comparison, which is never mentioned or described in the text. * Typo in caption of Figure 1 and the first line of section 3.3 Overall, I am hesitant to recommend the paper before cross-checking the issue with contamination proportion and learning more about how a VAE framework is indeed important for anomaly detection.
assistant
The paper proposes an improved extension of the Wasserstein auto-encoder for anomaly detection. The novelty is in proposing a weighted reconstruction error that penalizes the mapping of data with high reconstruction errors (mostly anomalies) into high probability regions. The idea being that an outlier would have a higher reconstruction error, and hence should be mapped to low-probability region of the latent distribution. Experimental Results: As a distribution mapping auto-encoder model, OP-DMA outperforms the deep learning based state-of-the-art models in the same domain. Overall Assessment: The authors have a nice idea of forcing the latent mappings of inputs to correlate with their reconstruction error. Overall, the method is promising, but I have the following concerns: * Using the reconstruction error as an anomaly score has been explored many years ago (check replicator neural networks), the novelty here is to enforce that on the latent space in the context of a variational auto-encoder. I am not sure if, from anomaly detection perspective, this is any better than simply using the reconstruction score. Why go the VAE route at all? * Is there a possibility that assuming a single multi-variate Gaussian, as a prior, too restrictive? Could it result in a high false alarm rate as well? I guess this could be answered by more experimental results on richer data sets (even synthetic is fine). * In most score based algorithms, the anomaly score is computed without assuming any prior knowledge about the contamination proportion. However, in the case of OP-DMA, the contamination parameter is used to train the auto-encoder that scores the data. This might result in an optimization that is very specific to the parameter setting. I strongly recommend a sensitivity analysis to study the robustness of the model against different values of contamination parameter. * Performance on synthetic data-set has not been presented. The set H in theorem 3 has not been defined. * Additional comparison with other non-distribution mapping state-of-the-art models such as LOF, oc-SVM, KNN would give a clearer idea of the performance. This is important, because in my past experience, non-deep learning methods give much better results on the benchmark data sets that the authors have evaluated their method on. In fact, a comparative analysis (See - https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152173) gives a very nice comparison. However, since the authors provide results using Avg F1-score, instead of AUC curve, it was not possible to compare them myself. * In figure 3, in the training process, the authors have describe to add the divergence between the latent and prior distribution to the loss function, however, nothing like this is clearly shown in the figure.The references of figures in the text are either out of place or incorrect. Figure 1(a) and (b) in reference to the text are incorrect. Figure 2 is the misleading figure as it doesn*t illustrate the anomaly detection process. Figure 3 has not been mentioned anywhere in the text. The authors have mentioned the comparison of their method with Wasserstien and variational auto-encoders in the text, while in table 2 and 4, AAE has also been shown as one of the method for comparison, which is never mentioned or described in the text. * Typo in caption of Figure 1 and the first line of section 3.3 Overall, I am hesitant to recommend the paper before cross-checking the issue with contamination proportion and learning more about how a VAE framework is indeed important for anomaly detection.