This paper presents the Variation Network (VarNet), a generative model providing means to manipulate the high-level attributes of a given input. The originality of our approach is that VarNet is not only capable of handling pre-defined attributes but can also learn the relevant attributes of the dataset by itself. These two settings can be easily combined which makes VarNet applicable for a wide variety of tasks. Further, VarNet has a sound probabilistic interpretation which grants us with a novel way to navigate in the latent spaces as well as means to control how the attributes are learned. We demonstrate experimentally that this model is capable of performing interesting input manipulation and that the learned attributes are relevant and interpretable.
This paper introduces a new framework for learning an interpretable representation of images and their attributes. The authors suggest decomposing the representation into a set of 'template' latent features, and a set of attribute-based features. The attribute-based features can be either 'free', i.e. discovered from the data, or 'fixed', i.e. based on the ground truth attributes. The authors encourage the decomposition of the latent space into the 'template' and the 'attributes' features by training a discriminator network to predict whether the attributes and the template features come from the same image or not.
While the idea is interesting, the paper is lacking an experimental section, so the methodology is impossible to evaluate. Furthermore, while the authors spend many pages describing their methodology, the writing is often hard to follow, so I am still confused about the exact implementation of the attribute features \phi(x, m) for example. The authors do point to the Appendix for their Experiments section, however this is not a good idea. The paper should be self-contained and the authors should not assume that their readers will read the information presented in the Appendix, which is always optional. 
Unfortunately, even the experimental section presented in the Appendix is not comprehensive enough to evaluate the proposed method. The authors train the model on a single dataset (MNIST), no baseline or ablation results are presented, and all the results are purely qualitative. Given that the ground truth attribute decomposition for MNIST is not known, even the qualitative results are impossible to evaluate. I recommend that the authors present quantitive results in the updated version of their paper (i.e. disentanglement metric scores, the log-likelihood of the reconstructions), including new experiments on a dataset like dSprites or CelebA, where the ground truth attributes are known.