It is challenging to disentangle an object into two orthogonal spaces of structure and appearance since each can influence the visual observation in a different and unpredictable way. It is rare for one to have access to a large number of data to help separate the influences. In this paper, we present a novel framework to learn this disentangled representation in a completely unsupervised manner. We address this problem in a two-branch Variational Autoencoder framework. For the structure branch, we project the latent factor into a soft structured point tensor and constrain it with losses derived from prior knowledge. This encourages the branch to distill geometry information. Another branch learns the complementary appearance information. The two branches form an effective framework that can disentangle object's structure-appearance representation without any human annotation. We evaluate our approach on four image datasets, on which we demonstrate the superior disentanglement and visual analogy quality both in synthesized and real-world data. We are able to generate photo-realistic images with 256 × 256 resolution that are clearly disentangled in structure and appearance.
Summary: The authors present an autoencoding strategy for disentangling structure and appearance in image data. They achieve this using a learned, spatial prior in the VAE framework.
Writing: The paper contains grammatical errors and I found Section 2 (2.1 and 2.2, specifically) to be a bit confusing as many ideas were described with words when they could have been outlined more precisely mathematically. 
Major comments:
The paper takes ideas from Zhang et al and Jakab et al and put them in a VAE context. The paper, however, constructs the ELBO in such a way that distance it from many key ideas of the VAE. Particularly, the paper decomposes the ELBO into three terms and proceed to define these terms as they wish. In this way, the novel contributions of this paper are left unclear and many decisions left unjustified.
- Incorporating landmark/spatial information into autoencoders is not a new idea. Zhang et al and Jakab et al both train autoencoders with disentangled structure, and Finn, 2016 [1] uses a similar spatial landmark strategy when learning representations. Incorporating structural information as a prior distribution (as in this paper) is an interesting idea. However, it is not clear that the defined prior log p(y) is a proper density (integrates to 1). Given the importance of a prior distribution in VAEs, this choice should be precisely justified (maybe relate it to beta-VAE or just use a properly normalizable distribution)
- The paper chooses variational distributions in such a way that removes the entropy term from the KL divergences. Specifically, both q(z | x, y) and p(z | y) have fixed, identity covariance, resulting in the KL divergences equating to L_2 distance. An important part of the VAE is the explicit incorporation of uncertainty by means of learned variances. Although this can sometimes be problematic (see beta-VAE), neglecting to include these variances at all removes an important aspect of VAE.
- The paper includes a likelihood model which also throws away key ideas from VAE. Although some neural likelihood models as in the VAE have issues with blurriness, the reasons are still not completely understood. However, there are well explored alternatives to what the authors propose. For example, many image-based VAEs use Bernoulli likelihoods [2, 3] or autoregressive likelihoods [4]. Autoregressive models, especially, can produce sharp images. The authors introduce a unnormalizable likelihood which combines L1 loss with a function that incorporates L1 distance in VGG space. Using VGG in the likelihood model is unjustified and seems unnecessarily complicated, given the existence of powerful decoders that already exist in VAE literature.
- The authors incorporate various connections and concatenations between neural networks and distributions that further complicate the variational lower bound. For example, p(z | y) is concatenated to q(z | x, y) in addition to acting as a prior on q(z | x, y). This introduces a dependency between the likelihood model and the variational posterior which normally does not exist. Furthermore skip connections are introduced between E_\theta and D_\theta, which complicate the ELBO further. The authors should explicitly write out the loss function they are optimizing at this point or describe how they are modifying ELBO to justify these chocise.
Overall, I find it difficult to call this a variational autoencoder given the liberal modifications to the evidence lower-bound. However, even if I was to interpret this work as an autoencoder with a custom loss function, this model ends up very similar to that in Zhang et al with the main differences being the inclusion of an equivariance constraint in Zhang that is not present in this paper and that Zhang et al use a feature map that’s multiplied by landmarks to incorporate appearance information whereas this paper uses the z representation as appearance information. 
The qualitative results of this paper, although good looking, are very similar to qualitative results in Zhang et al and Jakab et al. A quick comment: in Figure 6, you should use the same celebrity faces when comparing Jakab against your own work. The quantitative results only compare to the same model trained without the KL loss term; this, in my mind, is more of a sanity check than a fair baseline. The authors should be comparing against alternate strategies that incorporate spatial information, such as Zhang et al and Jakab et al.
[1] Finn, Chelsea, et al. "Deep spatial autoencoders for visuomotor learning." 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2016.
[2] Chen, Xi, et al. "Variational lossy autoencoder." arXiv preprint arXiv:1611.02731 (2016).
[3] http://ruishu.io/2018/03/19/bernoulli-vae/
[4] van den Oord, Aaron, et al. "Conditional image generation with pixelcnn decoders." Advances in Neural Information Processing Systems. 2016.