Anomaly Detection by Deep Direct Density Ratio Estimation
Estimating the ratio of two probability densities without estimating each density separately has been shown to provide useful solutions to various machine learning tasks such as domain adaptation, anomaly detection, feature extraction, and conditional density estimation. However, density ratio estimation in the context of deep learning has not been extensively explored yet. In this paper, we apply a Bregman-divergence minimization method for density ratio estimation to deep neural networks and investigate its properties and practical performance in image anomaly detection. Our numerical experiments on the CIFAR-10, CIFAR-100 and Fashion-MNIST datasets demonstrate that deep direct density ratio estimation greatly improves the anomaly detection ability and reduces the computation time over state-of-the-art methods.
The paper studies inlier-based outlier detection via density-ratio estimation in a deep learning context. Specifically, they consider the use of KLIEP in conjunction with a deep neural network. One finding is that batch normalisation is harmful for performance of this method. Experiments show consistent gains over a self-supervised baseline. The paper is generally well-written, and provides a nice overview of the density-ratio approach to anomaly detection. Its aims are clearly spelled out, and the careful derivation in Sec 3 of the Bregman perspective of inlier-based anomaly detection serves as a clean introduction to the topic. At the same time, the technical contribution of the paper is modest. As best I could tell, Sec 3 is entirely devoted to a (nice) review of the existing framework of (Sugiyama et al., 2012b), and contains no new material. Sec 4 seems to be the only novel technical contribution of the paper, namely, the observation that batch normalisation affects the KLIEP method adversely. This is certainly an interesting finding, but it is accompanied by no further critical analysis. In particular, there is no deeper explanation for why batch normalisation might affect KLIEP in this way; whether this finding is consistent for different choices of architecture; whether this finding is consistent for different choices of method (e.g., uLSIF); and so on. By itself, I feel there merely observing that batch norm interacts badly with KLIEP is not sufficiently deep as to carry the paper. The experiments compare KLIEP against an unsupervised baseline based on geometric transforms (GTs). This comparison is somehow unfair, since (as I understand) GTs assume one operates in a different problem setting altogether (where there is only a single sample of inliers, and no background contrast sample). Given this, it is surprising that GTs are able to be competitive with the proposed method in several cases; I feel this deserves more comment. It also raises the question of whether one may combine the proposed method with GTs. More such experiments might have made the paper stronger, but in their current form I don*t find them too illuminating or surprising. Another confusion that arose was that the experiments, as best I can tell, focus entirely on KLIEP. This makes the motivation for Sec 3 unclear -- the section introduces an abstract Bregman divergence view of anomaly detection, but then settles on one specific case of this which can be derived rather directly without this (as was done in the original KLIEP paper, to my knowledge). Had there been some study of different instantiations of the power divergence empirically, e.g., that might better justify the discussion in Sec 3. (I would imagine that for example KLIEP is more susceptible to outliers or label noise, which could motivate using intermediate values of ?.) Overall, while I appreciated the paper as an overview of inlier-based anomaly detection, its increment over the current literature appears modest. Minor comments: - *and it is used as an outlier score* -> *and is used as an outlier score* - *neural network ARCHITECTURES for complex image datasets* - *Deep SVDD approach ARE the distance of a data point* - why do you use p^* in Sec 3 rather than just p? - remove italics from Fig 1 labels. - *let us consider using A CNN to ESTIMATE the density* - it seems fairer to use the GT authors* implementation of their own method? - *cleverly impose a non-negativity constraint* -> omit the word *cleverly*.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Anomaly Detection by Deep Direct Density Ratio Estimation
Estimating the ratio of two probability densities without estimating each density separately has been shown to provide useful solutions to various machine learning tasks such as domain adaptation, anomaly detection, feature extraction, and conditional density estimation. However, density ratio estimation in the context of deep learning has not been extensively explored yet. In this paper, we apply a Bregman-divergence minimization method for density ratio estimation to deep neural networks and investigate its properties and practical performance in image anomaly detection. Our numerical experiments on the CIFAR-10, CIFAR-100 and Fashion-MNIST datasets demonstrate that deep direct density ratio estimation greatly improves the anomaly detection ability and reduces the computation time over state-of-the-art methods.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper studies inlier-based outlier detection via density-ratio estimation in a deep learning context. Specifically, they consider the use of KLIEP in conjunction with a deep neural network. One finding is that batch normalisation is harmful for performance of this method. Experiments show consistent gains over a self-supervised baseline. The paper is generally well-written, and provides a nice overview of the density-ratio approach to anomaly detection. Its aims are clearly spelled out, and the careful derivation in Sec 3 of the Bregman perspective of inlier-based anomaly detection serves as a clean introduction to the topic. At the same time, the technical contribution of the paper is modest. As best I could tell, Sec 3 is entirely devoted to a (nice) review of the existing framework of (Sugiyama et al., 2012b), and contains no new material. Sec 4 seems to be the only novel technical contribution of the paper, namely, the observation that batch normalisation affects the KLIEP method adversely. This is certainly an interesting finding, but it is accompanied by no further critical analysis. In particular, there is no deeper explanation for why batch normalisation might affect KLIEP in this way; whether this finding is consistent for different choices of architecture; whether this finding is consistent for different choices of method (e.g., uLSIF); and so on. By itself, I feel there merely observing that batch norm interacts badly with KLIEP is not sufficiently deep as to carry the paper. The experiments compare KLIEP against an unsupervised baseline based on geometric transforms (GTs). This comparison is somehow unfair, since (as I understand) GTs assume one operates in a different problem setting altogether (where there is only a single sample of inliers, and no background contrast sample). Given this, it is surprising that GTs are able to be competitive with the proposed method in several cases; I feel this deserves more comment. It also raises the question of whether one may combine the proposed method with GTs. More such experiments might have made the paper stronger, but in their current form I don*t find them too illuminating or surprising. Another confusion that arose was that the experiments, as best I can tell, focus entirely on KLIEP. This makes the motivation for Sec 3 unclear -- the section introduces an abstract Bregman divergence view of anomaly detection, but then settles on one specific case of this which can be derived rather directly without this (as was done in the original KLIEP paper, to my knowledge). Had there been some study of different instantiations of the power divergence empirically, e.g., that might better justify the discussion in Sec 3. (I would imagine that for example KLIEP is more susceptible to outliers or label noise, which could motivate using intermediate values of ?.) Overall, while I appreciated the paper as an overview of inlier-based anomaly detection, its increment over the current literature appears modest. Minor comments: - *and it is used as an outlier score* -> *and is used as an outlier score* - *neural network ARCHITECTURES for complex image datasets* - *Deep SVDD approach ARE the distance of a data point* - why do you use p^* in Sec 3 rather than just p? - remove italics from Fig 1 labels. - *let us consider using A CNN to ESTIMATE the density* - it seems fairer to use the GT authors* implementation of their own method? - *cleverly impose a non-negativity constraint* -> omit the word *cleverly*.
assistant
The paper studies inlier-based outlier detection via density-ratio estimation in a deep learning context. Specifically, they consider the use of KLIEP in conjunction with a deep neural network. One finding is that batch normalisation is harmful for performance of this method. Experiments show consistent gains over a self-supervised baseline. The paper is generally well-written, and provides a nice overview of the density-ratio approach to anomaly detection. Its aims are clearly spelled out, and the careful derivation in Sec 3 of the Bregman perspective of inlier-based anomaly detection serves as a clean introduction to the topic. At the same time, the technical contribution of the paper is modest. As best I could tell, Sec 3 is entirely devoted to a (nice) review of the existing framework of (Sugiyama et al., 2012b), and contains no new material. Sec 4 seems to be the only novel technical contribution of the paper, namely, the observation that batch normalisation affects the KLIEP method adversely. This is certainly an interesting finding, but it is accompanied by no further critical analysis. In particular, there is no deeper explanation for why batch normalisation might affect KLIEP in this way; whether this finding is consistent for different choices of architecture; whether this finding is consistent for different choices of method (e.g., uLSIF); and so on. By itself, I feel there merely observing that batch norm interacts badly with KLIEP is not sufficiently deep as to carry the paper. The experiments compare KLIEP against an unsupervised baseline based on geometric transforms (GTs). This comparison is somehow unfair, since (as I understand) GTs assume one operates in a different problem setting altogether (where there is only a single sample of inliers, and no background contrast sample). Given this, it is surprising that GTs are able to be competitive with the proposed method in several cases; I feel this deserves more comment. It also raises the question of whether one may combine the proposed method with GTs. More such experiments might have made the paper stronger, but in their current form I don*t find them too illuminating or surprising. Another confusion that arose was that the experiments, as best I can tell, focus entirely on KLIEP. This makes the motivation for Sec 3 unclear -- the section introduces an abstract Bregman divergence view of anomaly detection, but then settles on one specific case of this which can be derived rather directly without this (as was done in the original KLIEP paper, to my knowledge). Had there been some study of different instantiations of the power divergence empirically, e.g., that might better justify the discussion in Sec 3. (I would imagine that for example KLIEP is more susceptible to outliers or label noise, which could motivate using intermediate values of ?.) Overall, while I appreciated the paper as an overview of inlier-based anomaly detection, its increment over the current literature appears modest. Minor comments: - *and it is used as an outlier score* -> *and is used as an outlier score* - *neural network ARCHITECTURES for complex image datasets* - *Deep SVDD approach ARE the distance of a data point* - why do you use p^* in Sec 3 rather than just p? - remove italics from Fig 1 labels. - *let us consider using A CNN to ESTIMATE the density* - it seems fairer to use the GT authors* implementation of their own method? - *cleverly impose a non-negativity constraint* -> omit the word *cleverly*.