We propose negative sampling as an approach to improve the notoriously bad out-of-distribution likelihood estimates of Variational Autoencoder models. Our model pushes latent images of negative samples away from the prior. When the source of negative samples is an auxiliary dataset, such a model can vastly improve on baselines when evaluated on OOD detection tasks. Perhaps more surprisingly, we present a fully unsupervised version of employing negative sampling in VAEs: when the generator is trained in an adversarial manner, using the generator's own outputs as negative samples can also significantly improve the robustness of OOD likelihood estimates.
This paper discusses the detection of out-of-distribution (OOD) samples for variational autoencoders (VAE).
The idea is to train the encoder such that its output variational distribution q(z|\bar{x}) is pushed away from the prior of latent z. 
I think the paper needs more clarification and investigation for being published in the conference. 
My major concern is that more empirical investigation is necessary since the formulation provides a minor novelty. 
Specific points are given below. 
1) Weak novelty in terms of model design. 
The objective function consists of the standard (negative) ELBO term and additional KL term to modify the variational posterior of negative samples. 
This modification can be regarded as a form of outlier exposure (Hendrycks et al. 2018) specialized for VAE. 
The choice of \bar{p} is not much investigated. 
Any discussion if we use a more sophisticated model such as VampPrior* for stronger modeling capacity. 
* J. Tomczak and M. Welling, VAE with a VampPrior, AISTATS 2018.
2) The use generated samples as negative samples is interesting but mysterious. 
The authors conjecture that this works because the generated samples come from near the data manifold, but in-distribution samples and negative samples can be indistinguishable when the generative model is very well trained. 
What happens if, for example, the negative samples are generated by data augmentation techniques (such as cropping, rotation, mirroring, though mirroring and much rotation may be unsuitable for text images)? 
This can also produce near-manifold points. 
A deeper analysis why generated samples can improve the OOD detection performance is necessary. 
Furthermore, why does not this approach impact much for color images in Table 4?
3) More details of experimental procedures. 
3-1) How was data points are generated from VAE as negative samples? 
Possible ways are:
* sample z ~ p(z), then draw from the decoder x ~ p(x|z).
* use negative prior z ~ \bar{p}(z), then draw from the decoder x ~ p(x|z).
* this seems weird: use variational posterior z ~ q(z|x), then x ~ p(x|z).
3-2) Latent dimension of 10 for grayscale images seems small. 
Does the size affect the OOD detection performance when the size is 50 or 100 to make the model richer. 
3-3) How was the variance obtained when the decoder uses the Gaussian likelihood?
* fixed value?
* learned for each pixel?
* output from the decoder?
4) If we have access to diverse negative datasets, can the ODD detection perform better? 
Mixing multiple datasets or using both available dataset and generated samples can improve the performance while the test OOD samples are kept unseen. 
For example, train VAE on MNIST while using KMNIST and EMNIST as the negative sets to detect Fashion-MNIST as ODD.