VARIATIONAL AUTOENCODER WITH ARBITRARY CONDITIONING
We propose a single neural probabilistic model based on variational autoencoder that can be conditioned on an arbitrary subset of observed features and then sample the remaining features in "one shot". The features may be both real-valued and categorical. Training of the model is performed by stochastic variational Bayes. The experimental evaluation on synthetic data, as well as feature imputation and image inpainting problems, shows the effectiveness of the proposed approach and diversity of the generated samples.
The goal of this paper is to use deep generative models for missing data imputation. This paper proposes learning a latent variable deep generative model over every randomly sampled subset of observed features. First, a masking variable is sampled from a chosen prior distribution. The mask determines which features are observed. Then, the likelihood of the observed features is maximized via a lower bound. Inference in this latent variable model is achieved through the use of an inference network which conditions on the set of "missing" (to the generative model) features.
Novelty:
Generative models have a long history of being used to impute missing data. e.g. http://www.cs.toronto.edu/~fritz/absps/ranzato_cvpr2011.pdf, https://arxiv.org/pdf/1610.04167.pdf,
https://arxiv.org/pdf/1808.01684.pdf, https://arxiv.org/pdf/1401.4082.pdf [Appendix F]
It is a little difficult to guage what the novelty of this work is.
Clarity
This is a poorly written paper. Distilling the proposed methodology down to one paragraph was challenging since the text meanders through several concepts whose relevance to the overarching goal is questionable. For example, it is not clear what Section 3.2 adds to the discussion. The text describes a heuristic used in learning GSNNs only to say that the loss function used by GSNNs is not used in the experimental section for this paper -- this renders most of 4.3.2 redundant. There are issues like awkward grammar, sloppy notation, and spelling mistakes (please run spell check!) throughout the manuscript. Please use a different notation when referring to the variational distributions (do not re-use "p").
Experimental Results
The model is evaluated against MICE and MissForest on UCI datasets. RMSE and accuracy of classification (from imputed data is compared). The complexity of data considered is simplistic (and may not make use of the expressivity of the deep generative model). Why not run these experiments on datasets like MNIST and Omniglot?
Beyond that:
(a) was there any comparison to how classification performance behaves when using another neural network based imputation baseline (e.g. the method in Yoon et. al)?
(b) the *kind* of missingness considered here appears to be MCAR (the easiest kind to tackle) -- did you consider experiments with other kinds of missingess?
The qualitative results presented in this work are interesting. The method does appear to produce more diverse in-paintings than the method from Yeh et. al (though the examples considered are not aligned).
Table 5 claims negative log-likelihood numbers on MNIST as low as 61 and 41 (I assume nats...). These numbers do not make sense. How were they computed?
Priors on b:
What kind of priors on b did you experiment with?