Despite their successes, deep neural networks still make unreliable predictions when faced with test data drawn from a distribution different to that of the training data, constituting a major problem for AI safety. While this motivated a recent surge in interest in developing methods to detect such out-of-distribution (OoD) inputs, a robust solution is still lacking. We propose a new probabilistic, unsupervised approach to this problem based on a Bayesian variational autoencoder model, which estimates a full posterior distribution over the decoder parameters using stochastic gradient Markov chain Monte Carlo, instead of fitting a point estimate. We describe how information-theoretic measures based on this posterior can then be used to detect OoD data both in input space as well as in the model's latent space. The effectiveness of our approach is empirically demonstrated.
This paper studies the problem of out-of-distribution data detection, which is an important problem in machine learning. The authors propose to use Bayesian variational autoencoder which applies SGHMC to get samples of the weights of the encoder and the decoder. The proposed method is tested on two benchmarks to demonstrate effectiveness.
The proposed Bayesian variational autoencoder appears to be technically sound. When applying it to OoD detection, effective sample size is used to quantify how much the posterior changes given the new data. The authors claim that ESS will be large when the data is in-distribution since all samples explain the data equally well. First, I’m not sure this is true that all samples from the posterior should explain the data equally well even if it is in-distribution. Second, if the data is out of distribution, it is likely that all samples explain the data equally bad which also results in high ESS. In practice, it is very likely that p(x*|theta) are low for all the theta when x* is out-of-distribution. Am I missing something here?
How to determine whether a data is out-of-distribution or not based on ESS? Is the threshold of ESS a hyperparameter to tune?
For the experiments, I wonder why the authors put Gamma hyper priors for BVAE which was not used in the previous work that use SGHMC. Is there any reason for doing this? Again, it is unclear to me how the authors decide whether a data is out-of-distribution or not based on ESS.
It seems like simply applying SGHMC for the decoder parameters is sufficient, as the other treatments only improve the results incrementally but adding large computational and storage cost. I’m not familiar with the literature enough to tell whether the results of previous methods are reasonable or not. By looking at the table, it seems that the proposed method achieves some gain over the previous methods. 
In the experiments, BVAE only keeps the most recent 10 samples. Aren’t the samples very similar? Since the thinning interval is only 1 epoch.
It would make the paper stronger if the authors are able to demonstrate the usefulness of detecting OoD in latent space through experiments.