VARIATIONAL INFERENCE OF DISENTANGLED LATENT CONCEPTS FROM UNLABELED OBSERVATIONS
Disentangled representations, where the higher level data generative factors are reflected in disjoint latent dimensions, offer several benefits such as ease of deriving invariant representations, transferability to other tasks, interpretability, etc. We consider the problem of unsupervised learning of disentangled representations from large pool of unlabeled observations, and propose a variational inference based approach to infer disentangled latent factors. We introduce a regularizer on the expectation of the approximate posterior over observed data that encourages the disentanglement. We also propose a new disentanglement metric which is better aligned with the qualitative disentanglement observed in the decoder's output. We empirically observe significant improvement over existing methods in terms of both disentanglement and data likelihood (reconstruction quality).
########## UPDATED AFTER AUTHOR RESPONSE ##########
Thanks for the good revision and response that addressed most of my concerns. I am bumping up my score. 
###############################################
This paper presents a Disentangled Inferred Prior (DIP-VAE) method for learning disentangled features from unlabeled observations following the VAE framework. The basic idea of DIP-VAE is to enforce the aggregated posterior q(z) = E_x [q(z | x)] to be close to an identity matrix as implied by the commonly chosen standard normal prior p(z). The authors propose to moment-match q(z) given it is hard to minimize the KL-divergence between q(z) and p(z). This leads to one additional term to the regular VAE objective (in two parts, on- and off-diagonal). It has the similar property as beta-VAE (Higgins et al. 2017) but without sacrificing the reconstruction quality. Empirically the authors demonstrate that DIP-VAE can effectively learn disentangled features, perform comparably better than beta-VAE and at the same time retain the reconstruction quality close to regular VAE (beta-VAE with beta = 1). 
The paper is overall well-written with minor issues (listed below). I think the idea of enforcing an aggregated (marginalized) posterior q(z) to be close to the standard normal prior p(z) makes sense, as opposed to enforcing each individual posterior q(z|x) to be close to p(z) as (beta-)VAE objective suggests. I would like to make some connection to some work on understanding VAE objective (Hoffman & Johnson 2016, ELBO surgery: yet another way to carve up the variational evidence lower bound) where they derived something along the same line of an aggregated posterior q(z). In Hoffman & Johnson, it is shown that KL(q(z) | p(z)) is in fact buried in ELBO, and the inequality gap in Eq (3) is basically a mutual information term between z and n (the index of the data point). Similar observations have led to the development of VAMP-prior (Tomczak & Welling 2017, VAE with a VampPrior). Following the derivation in Hoffman & Johnson, DIP-VAE is basically adding a regularization parameter to the KL(q(z) | p(z)) term in standard ELBO. I think this interpretation is complementary to (and in my opinion, more clear than) the one that’s described in the paper. 
My concerns are mostly regarding the empirical studies: 
1. One of my main concern is on the empirical results in Table 1. The disentanglement metric score for beta-VAE is suspiciously lower than what’s reported in Higgins et al., where they reported a 99.23% disentanglement metric score on 2D shape dataset. I understand the linear classier is different, but still the difference is too large to ignore. Hence my current more neutral review rating. 
2. Regarding the correlational plots (the bottom row of Table 3 and 4), I don’t think I can see any clear patterns (especially on CelebA). I wonder what’s the point of including them here and if there is a point, please explain them clearly in the paper. 
3. Figure 2 is also a little confusing to me. If I understand the procedure correctly, a good disentangled feature would imply smaller correlations to other features (i.e., the numbers in Figure 2 should be smaller for better disentangled features). However, looking at Figure 2 and many other plots in the appendix, I don’t think DIP-VAE has a clear win here. Is my understanding correct? If so, what exactly are you trying to convey in Figure 2? 
Minor comments: 
1. In Eq (6) I think there are typos in terms of the definition of Cov_q(z)(z)? It appears as only the second term in Eq (5). 
2. Hyperparameter subsection in section 3: Shouldn’t \lambda_od be larger if the entanglement is mainly reflected in the off-diagonal entries? Why the opposite? 
3. Can you elaborate on how a running estimate of Cov_p(x)(\mu(x)) is maintained (following Eq (6)). It’s not very clear at the current state of the paper. 
4. Can we have error bars in Table 2? Some of the numbers are possibly hitting the error floor. 
5. Table 5 and 6 are not very necessary, unless there is a clear point.