Learning disentangled representations of data is one of the central themes in unsupervised learning in general and generative modeling in particular. In this work, we tackle a slightly more intricate scenario where the observations are generated from a conditional distribution of some known control variate and some latent noise variate. To this end, we present a hierarchical model and a training method (CZ-GEM 1 ) that leverages some of the recent developments in likelihoodbased and likelihood-free generative models. We show that CZ-GEM introduces the right inductive biases that ensure the disentanglement of the control from the noise variables, while also keeping the components of the control variable disentangled. This is achieved without compromising on the quality of the generated samples. Our approach is simple, general, and can be applied both in the supervised and unsupervised settings.
This paper proposes a method for learning disentangled representations.  The approach is used on both supervised (where the factors to be disentangled are known) and unsupervised settings. The authors demonstrate the efficacy of their approach in both settings on several datasets with both quantitative and qualitative results.
This task is an important one. However, I found that the contribution of this paper is fairly small. The proposed approach seems reasonable but it is mostly a work of engineering and provides little insights into the problem nor the proposed model.
The setup where labeled data (c) also seems a bit unnatural (this also seems to be confirmed by the fact that the authors had to build datasets for the problem). Perhaps the authors could give examples of situations where this would naturally arise. In practice, it seems difficult to obtain these data for all required variables to be disentangled.
The unsupervised results are more interesting but not very much explored (a single set of sampled faces). I was also curious as to why the learned Y's are blurry. This sort of two-stage generation is also potentially interesting, I was wondering if the authors had ideas to generalize this idea. 
I also was not convinced by the experiments which are mostly qualitative. I did not find that this set of experiments provide enough support to the proposed method.
Detailed comments:
- It is a bit unclear to me how the authors propose to obtain independent posteriors over z and c. Is it purely empirical or is there a formal reason that guarantees it?
- Some of the figures your report are compelling but it is a bit unclear to the reader if the results are general (e.g., the examples could have been hand-picked). Are there any quantitative measures you could provide (in addition to Tables 1 and 2 which don't measure the quality of the approach)?
- Comparing to CGAN seems reasonable but given the task at hand, it seems like other methods could have been tried (although I do realize that no one may have done this before for deep generative models).
Other comments:
- In Figure 3, it would be good to label the upper trapezoid.
- Some paragraphs are very long and the manuscript may benefit from segmenting them into multiple paragraphs.