Recent work has shown increased interest in using the Variational Autoencoder (VAE) framework to discover interpretable representations of data in an unsupervised way. These methods have focussed largely on modifying the variational cost function to achieve this goal. However, we show that methods like β-VAE simplify the tendency of variational inference to underfit causing pathological overpruning and over-orthogonalization of learned components. In this paper we take a complementary approach: to modify the probabilistic model to encourage structured latent variable representations to be discovered. Specifically, the standard VAE probabilistic model is unidentifiable: the likelihood of the parameters is invariant under rotations of the latent space. This means there is no pressure to identify each true factor of variation with a latent variable. We therefore employ a rich prior distribution, akin to the ICA model, that breaks the rotational symmetry. Extensive quantitative and qualitative experiments demonstrate that the proposed prior mitigates the trade-off introduced by modified cost functions like β-VAE and TCVAE between reconstruction loss and disentanglement. The proposed prior allows to improve these approaches with respect to both disentanglement and reconstruction quality significantly over the state of the art.
This paper presents a methodology to bring together independent subspace analysis and variational auto-encoders. Naturally, in order to do that, the authors propose a specific family of prior distributions that lead to subspace independence the Lp-nested distribution family. This prior distribution is then used to learn disentangled and interpretable representations. The mutual information gap is taken as the measure of disentanglement, while the reconstruction loss measures the quality of the representation. Experiments on the sPrites dataset are reported, and comparison with the state of the art shows some interesting results.
I understand the limitations of current approaches for learning disentangled representations, and therefore agree with the motivation of the manuscript, and in particular the choice of the prior distribution. However, I did not find the answer to some important questions, and generally speaking I believe that the contribution is not completely and clearly described.
P1) What is the shape of the posterior distribution?
P2) How does the reparametrization trick work in your case?
P3) How can one choose the layout of the subspaces, or this is also learned?
Moreover, and this is crucial, the proposed method is not clearly explained. Different concepts are discussed, but there is no summary and discussion of the proposed method as a whole. The reader must infer how the method works from the different pieces. 
When discussing the performance of different methods, and even if in the text the four different alternatives are clearly explained, in figure captions and legens the terminology changes (ISA-VAE, ISA-beta-VAE, beta-VAE, beta-ISA-VAE, etc). This makes the discussion very difficult to follow, as we do not understand which figures are comparable to which, and in which respect.
In addition, there are other (secondary) questions that require an answer.
S1) After (10) you mention the subspaces v_1,...v_l_o. What is the formal definition of these subspaces?
S2) The definition of the distribution associated to ISA also implies that n_i,k = 1 for all i and k, right?
S3) Could you please formally write the family of distributions, since applying this to a VAE is the main contribution of your manuscript?
S4) Which parameters of this family are learned, and which of them are set in advance?
S5) From Figure 4 and 5, I understand that the distributions used are of the type in (7) and not (10). Can you comment on this?
S6) How is the Lp layout chosen?
S7) Why the Lp layout for ISA-beta-VAE in Figure 5 is not the same as in Figure 4 for ISA-VAE?
S8) What are the plots in Figure 4? They are difficult to interpret and not very well discussed.
Finally, there are a number of minor corrections to be made.
Abstract: latenT
Equation (3) missig a sum over j
Figure 1 has no caption
In (8), should be f(z) and not x.
Before (10), I understand you mean Lp-nested
I did not find any reference to Figure 3
In 4.1, the standard prior and the proposed prior should be referred to with different notations.
For all these reasons I recommend to reject the paper, since in my opinion it is not mature enough for publication.