Likelihood-based generative models are a promising resource to detect out-ofdistribution (OOD) inputs which could compromise the robustness or reliability of a machine learning system. However, likelihoods derived from such models have been shown to be problematic for detecting certain types of inputs that significantly differ from training data. In this paper, we pose that this problem is due to the excessive influence that input complexity has in generative models' likelihoods. We report a set of experiments supporting this hypothesis, and use an estimate of input complexity to derive an efficient and parameter-free OOD score, which can be seen as a likelihood-ratio, akin to Bayesian model comparison. We find such score to perform comparably to, or even better than, existing OOD detection approaches under a wide range of data sets, models, model sizes, and complexity estimates.
# ==== Summary of the paper ====
This paper proposes a new criterion for out-of-distribution (OOD) detection for
generative modelling of images (absence of labels). The OOD detection task is defined as follows: given a generative model p(x) trained on some data $X \sim p^*$ (where $p^*$ is the true unknown data generating distribution), and a test point y, how can we make use of p to detect that y is not drawn from $p^*$ (i.e., out of distribution)? The first natural idea is to use the likelihood (density) given by p, which, by now, is known to be problematic since the likelihood can be high when evaluated on data points from a completely different domain. 
This paper contributes the following results:
1. Show that "simple" images (e.g., constant color) tend to give high likelihood, whereas complex images (e.g., noise) tend to give low likelihood. See Figure 2. The paper further quantifies this complexity with the normalized size of the compressed input images, as given by a compression algorithm. The paper shows that there is a negative correlation between the compressed size and the likelihood (models trained on CIFAR 10. See Figure 4).
2. To address this issue with using only the likelihood for OOD detection, the paper proposes a new measure $S(x)$ given by the negative log likelihood minus the normalized compressed size. See Eq 1. The paper connects this measure to Bayesian model selection.
Empirical results on more than 10 image datasets show that the proposed measure works better than the negative log likelihood in most cases.
# ==== Review ====
The paper is easy to follow. The finding that simple images tend to give high likelihood is interesting. My concerns are:
1. How much does the conclusion that "simple images tend to give high likelihood" depend on the complexity of the model? As far as I can see, only a few models are studied here: Glow and PixelCNN++. What is the quality of the learned models? Do they generate realistic images? What happens if you consider simpler models (say, reduce the number of layers in Glow.)?
2. Given a model p and a test input image y, how exactly do you tell if y is out of distribution? Is there a threshold? If so, what is the threshold? I understand that by AUROC used in Table 1, you vary the threshold. For each value of the threshold, you compute the true positive and false positive rates, and plot the ROC curve. The reported numbers in Table 1 are areas under the curve. Is this correct? But this does not explain how to perform OOD detection given one input image.
3. What is the reason for the poor AUROC in the case of TinyImageNet in Table 1? This is an interesting case since it may suggest that there are other hidden factors (besides the complexity of images) that can affect the OOD detection with the likelihood.
4. Have you tried to run the experiment in Table 1 with a model trained on Constant (simplest images) or Noise (most complex images)?  What happens? Also, why not also include CIFAR10 (used to train the model) in the list of datasets in Table 1?
Overall, my main concerns are with the thoroughness of the experiments, and that the two main contributions (summarized above) may not be enough. I will consider my evaluation again after seeing responses from the authors.
# ==== Minor. Did not affect the score ====
* The bottom margin seems off. Please check whether the Latex template is used correctly.
* The paragraph before Section 3: "... could replace the computed likelihood values for the negative of our complexity estimate ..." I think it is too soon to make this conclusion. 
* The term "likelihood ratio test" used in Section 3.2 is very misleading. There is no hypothesis testing there.
* Might be better to briefly describe the meaning of AUROC at the beginning of section 5.