DEEP ANOMALY DETECTION WITH OUTLIER EXPOSURE
It is important to detect anomalous inputs when deploying machine learning systems. The use of larger and more complex inputs in deep learning magnifies the difficulty of distinguishing between anomalous and in-distribution examples. At the same time, diverse image and text data are available in enormous quantities. We propose leveraging these data to improve deep anomaly detection by training anomaly detectors against an auxiliary dataset of outliers, an approach we call Outlier Exposure (OE). This enables anomaly detectors to generalize and detect unseen anomalies. In extensive experiments on natural language processing and small-and large-scale vision tasks, we find that Outlier Exposure significantly improves detection performance. We also observe that cutting-edge generative models trained on CIFAR-10 may assign higher likelihoods to SVHN images than to CIFAR-10 images; we use OE to mitigate this issue. We also analyze the flexibility and robustness of Outlier Exposure, and identify characteristics of the auxiliary dataset that improve performance.
I have read authors' reply.  In response to authors' comprehensive reply and feedback. I upgrade my score to 6. As authors mentioned, the extension to density estimators is an original novelty of this paper, but I still have some concern that OE loss for classification is basically the same as [2]. I think it is better to clarify this in the draft. 
Summary===
This paper proposes a new fine-tuning method for improving the performance of existing anomaly detectors. The main idea is additionally optimizing the “Outlier Exposure (OE)” loss on outlier dataset. Specifically, for softmax classifier, the authors set the OE loss to the KL divergence loss between posterior distribution and uniform distribution. For density estimator, they set the OE loss to a margin ranking loss. The proposed method improves the detection performance of baseline methods on various vision and NLP datasets. While the research topic of this paper is interesting, I recommend rejections because I have concerns about novelty and the experimental results.
Detailed comments ===
1. OE loss for softmax classifier
For softmax classifier, the OE loss forces the posterior distribution to become uniform distribution on outlier dataset. I think this loss function is very similar to a confidence loss (equation 2) proposed in [2]: Lee et al., 2017 [2] also proposed the loss function minimizing the KL divergence between posterior distribution and uniform distribution on out-of-distribution, and evaluated the effects of it on "unseen" out-of-distribution (see Table 1 of [2]). Could the authors clarify the difference with the confidence loss in [2], and compare the performance with it? Without that, I feel that the novelty of this paper is not significant.
2. More comparison with baselines
The authors said that they didn’t compare the performance with simple inference methods like ODIN [3] since ODIN tunes the hyper-parameters using data from (tested) out-of-distribution. However, I think that the authors can compare the performance with ODIN by tuning the hyper-parameters of it on outlier dataset which is used for training OE loss. Could the authors provide more experimental results by comparing the performance with ODIN? 
3. Related work
I would appreciate if the authors can survey and compare more baselines such as [4] and [5]. 
[1] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. International Conference on Learning Representations, 2017. 
[2] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. International Conference on Learning Representations, 2018. 
[3] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. International Conference on Learning Representations, 2018. 
[4] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. In NIPS, 2018.
[5] Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L. Willke. Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers, In ECCV, 2018.