Learning Disentangled Representations with Semi-Supervised Deep Generative Models
Variational autoencoders (VAEs) learn representations of data by jointly training a probabilistic encoder and decoder network. Typically these models encode all features of the data into a single variable. Here we are interested in learning disentangled representations that encode distinct aspects of the data into separate variables. We propose to learn such representations using model architectures that generalise from standard VAEs, employing a general graphical model structure in the encoder and decoder. This allows us to train partially-specified models that make relatively strong assumptions about a subset of interpretable variables and rely on the flexibility of neural networks to learn representations for the remaining variables. We further define a general objective for semi-supervised learning in this model class, which can be approximated using an importance sampling procedure. We evaluate our framework's ability to learn disentangled representations, both by qualitative exploration of its generative capacity, and quantitative evaluation of its discriminative ability on a variety of models and datasets.
This paper investigates the use of models mixing ideas from 'classical' graphical models (directed graphical models with nodes with known meaning, and with known dependency structures) and deep generative models (learned inference network, learned factors implemented as neural networks). A particular focus is the use of semi-supervised data and structure to induce 'disentangling' of features.  They test on a variety of datasets and show that disentangled features are interpretable and can be used for downstream tasks.
In contrast with recent work on disentangling ("On the Emergence of Invariance and Disentangling in Deep Representations", "Early visual concept learning with unsupervised deep learning", and others), here the disentangling is not really an emergent property of the model's inductive bias, and more related to the (semi)-supervision provided on semantic variables of the model.
My opinion on the paper is overall positive - the derivations are correct, the model and ideas contribute to the intersection of graphical models / probabilistic modeling and deep learning, and well-made/detailed experiments support the authors' claims.
Negative:
- I don't think the intersection of ideas in structured graphical models and deep generative model is particularly novel at this point (see many of the paper's own references). In particular I am not convinced by the claims of increased generality. 
Clearly there is a variety of deep generative models that fall in the PGM-DL spectrum described by the authors and performing inference with flexible set of observed variables mostly involves using different inference networks (with potential parameter reuse for efficiency). Regarding continuous domain latents: I don't see what would prevent the semi-supervised model from Kingma et al from handling continous latent variable: wouldn't simply involve changing q(y|x)  to a continuous distribution and reparametrize (just as they do for the never-supervised z)? Generally, methodologically the papers appears somewhat incremental (including the automated construction of the stochastic computation graph, which is similar to what something like Edward (Tran et al.) does).
- Some numerical experiments results are slightly disappointing (the semi-supervised MNIST numbers seem worse than Kingma et al.; the multi-MNIST reconstructions are somewhat blurry; what do samples look like?)
Positive:
- The paper is well written and tells a compelling story. 
- The auxiliary bound introduced in (4)-(5) is novel and and allows the posterior in the semi-supervised case to force learning over the classification network (in corresponding equation (6) of Kingma et al. the classification network does not appear). Do the authors know how the variance between (4) and the analogous expression from Kingma et al compare?
- The rest of the  experiments (figure 2, 5) paint a clear picture of why having semantics inform the structure of the generative model can lead to interesting way to probe the behavior of our generative models.