Among deep generative models, flow-based models, simply referred as flows in this paper, differ from other models in that they provide tractable likelihood. Besides being an evaluation metric of synthesized data, flows are supposed to be robust against out-of-distribution (OoD) inputs since they do not discard any information of the inputs. However, it has been observed that flows trained on FashionMNIST assign higher likelihoods to OoD samples from MNIST. This counterintuitive observation raises the concern about the robustness of flows' likelihood. In this paper, we explore the correlation between flows' likelihood and image semantics. We choose two typical flows as the target models: Glow, based on coupling transformations, and pixelCNN, based on autoregressive transformations. Our experiments reveal surprisingly weak correlation between flows' likelihoods and image semantics: the predictive likelihoods of flows can be heavily affected by trivial transformations that keep the image semantics unchanged, which we call semantic-invariant transformations (SITs). We explore three SITs (all small pixel-level modifications): image pixel translation, random noise perturbation, latent factors zeroing (limited to flows using multi-scale architecture, e.g. Glow). These findings, though counter-intuitive, resonate with the fact that the predictive likelihood of a flow is the joint probability of all the image pixels. So flows' likelihoods, modeling on pixel-level intensities, is not able to indicate the existence likelihood of the high-level image semantics. We call for attention that it may be abuse if we use the predictive likelihoods of flows for OoD samples detection.
Summary:
The paper studies likelihood-based models of images, such as Glow and PixelCNN. The paper shows empirically that image transformations that preserve semantics (e.g. translations by a few pixels) produce images that have lower probability (density) under such models.
Decision:
The paper is studying an important topic, which is how to use likelihood-based models correctly, and it warns against the misuse of such models. I agree with the paper's conclusion that we should be careful when using likelihood-based models.
Nonetheless, in my opinion the paper is a clear reject. The paper is full of flaws, incorrect statements, poorly constructed arguments, speculative explanations, and superficial descriptions of previous work.
Broadly, the main issue with the paper is the following:
(a) It begins with flawed assumptions about how a likelihood-based model is expected to behave.
(b) It tests two likelihood-based models experimentally and finds that they don't behave according to the assumptions.
(c) It concludes that we need to be cautious when using likelihood-based models.
As I said, I agree that we should be careful when using likelihood-based models, but I worry that the way the paper reaches this conclusion can mislead and misinform readers. In what follows, I elaborate on specific issues with the paper in more detail.
Issue #1:
The main flaw of the paper is the assumption that semantic-preserving transformations shouldn't reduce the likelihood of the model (beginning of section 4). This is incorrect. To see why, consider a semantic-preserving transformation x' = T(x). As defined in the paper, a semantic-preserving transformation is one that doesn't change the label y of an image x. This can be formalized as:
p(y | x') = p(y | x)
By Bayes' rule, from the above it follows that:
p(x' | y) p(y) / p(x') = p(x | y) p(y) / p(x) 
=> p(x' | y) / p(x') = p(x | y) / p(x)
Clearly p(x') can be different from p(x), as long as p(x' | y) is different from p(x | y) by the same factor. Hence, it doesn't follow that if p(y | x') = p(y | x) then p(x') = p(x), which is what the paper incorrectly assumes. To be clear, in the above expressions, p() refers to the true data distribution and not to a model that approximates it.
For example, consider images of digits, where the digit is generally in the centre of the image. Moving a digit to the corner will result in a less likely image, because it's unlikely that digits appear in corners. However, it won't change the classification of that digit, since all digits are less likely to appear in corners in exactly the same way. In fact, this is exactly why we see in section 4.1 that the likelihood of the model decreases as the image is translated to the left; the model is behaving exactly as it is supposed to.
Similarly, in section 4.2 where noise is added to the image, the model is again behaving exactly as it is supposed to. Adding Gaussian noise to the image results in a distribution p(x') that is equal to convolving the original distribution p(x) with the noise distribution p(noise) which is an isotropic Gaussian. As a result, p(x') will be a more diffuse version of p(x), hence samples x' will have on average low probability (density) under p(x), exactly as expected, and exactly as the experiment observes.
Issue #2:
The paper incorrectly assumes that out-of-distribution examples should have low probability (density). This is incorrect, and a common misconception that results from confusing high probability with typicality. In fact, out-of-distribution examples can have high probability (density). Here are two examples that illustrate that:
Suppose you flip a bent coin a million times, with 10% probability of the coin coming up heads. The in-distribution samples (the typical set) are those sequences of coin tosses that have roughly 100 thousand heads. However, the most likely outcome is the all-tails sequence. This outcome is clearly atypical, and many people would agree that it's out-of-distribution, but it has the highest probability.
Consider one million independent Gaussian variables, each with mean 0 and variance 1. Due to the law of large numbers, a typical draw of these variables will have average squared value very close to 1, hence the outcome of all variables being zero is very atypical and many people would agree it's out-of-distribution. However, the all-zero outcome is in fact the one with the highest probability density.
Given the above, the following two statements copied from the paper are flawed, and potentially misleading:
"The foundation of using likelihood-based models for OoD detection is that they are supposed to assign much lower likelihoods for OoD samples than in-distribution samples"
"In OoD detection, we assume that a sample with a higher likelihood indicates that it is more likely to be an in-distribution sample"
Fundamentally, I think the issue is that the paper incorrectly assumes that all images with the same semantics (e.g. all images of the digit 3) must be in-distribution. However this is not necessarily true. For example, the true data-generating process of MNIST images (i.e. asking people to write down a digit, scanning it, denoising it, cropping it and centring it) is unlikely to produce images where the digit is not in the centre or the background is noisy. Hence, the images considered in sections 4.1 and 4.2 are indeed out-of-distribution with respect to the true data distribution of MNIST, and are not adversarial examples of the models as the paper suggests.
Further issues:
The paper is ostensibly about flow models, but in fact very little is specific to flow models, and most of the discussion, where correct, applies to likelihood-based models in general. In fact, PixelCNN is not a flow model, even though the paper misleadingly describes it as such. PixelCNN can be used to model discrete random variables, whereas flow models are used for continuous random variables (flows for discrete random variables exist, but they are different from PixelCNN). That said, if PixelCNN is used to model continuous random variables then by reparameterization it can be viewed as a flow model with one layer, but that would be an unusual way to present it. Same for WaveNet.
"It is also believed that flows can be used to detect out-of-distribution(OoD) samples by assigning low likelihoods on them."
Believed by whom? A citation is needed here.
"Flows can roughly be divided into two categories [coupling flows and autoregressive flows]"
There are several flows that fall in neither of these categories, such as linear flows, residual flows, planar flows, radial flows, Sylvester flows, neural ODEs, FFJORD, and many others.
"The autoregressive property of an autoregressive layer is enforced by masking."
There are other ways of enforcing the autoregressive property (e.g. RNNs); masking is just one of them.
Eq. (5) is not correct in general. The bits per dimension should be approximated by:
BPD = (NLL - log|B|) / ((h x w x c) * log2)
where |B| is the quantization volume, provided |B| is small. That is, if each pixel with range [0, 1] is quantized into 10 bins, then |B| = 0.1 ^ (h x w x c).
"This surprising difference can be attributed to the difference of their architectures" (beginning of page 5)
The explanation that follows is speculative, but is not presented as such, which can be misleading.
"We may reasonably suspect that flows’ counter-intuitive likelihood assignment is dominated by the inherent differences of pixel-level statistics associated to the image semantics"
This is also speculation.
"PixelCNN is more sensitive to the noises, because its pixel-wise modeling quickly augment and propagate the influences of the added noise"
Also speculation.
"We find that the semantic object of a test image depends heavily on the last factored latent zL, rather than the preceding factors"
As far as I can see, there isn't evidence in support of that statement in the paper.
"Considering the weak correlation between flows’ likelihoods and image semantics, it is inappropriate to use them for OoD samples detection"
Given the flawed assumption about the role of image semantics, I don't think there is evidence for that.
"In terms of image generation, we expect that every single generated pixel in a image is the most likely one"
This is inaccurate; when generating images from a model, we don't get the most likely pixels, but samples from the joint distribution over pixels.