The variational autoencoder, one of the generative models, defines the latent space for the data representation, and uses variational inference to infer the posterior probability. Several methods have been devised to disentangle the latent space for controlling the generative model easily. However, due to the excessive constraints, the more disentangled the latent space is, the lower quality the generative model has. A disentangled generative model would allocate a single feature of the generated data to the only single latent variable. In this paper, we propose a method to decompose the latent space into basis, and reconstruct it by linear combination of the latent bases. The proposed model called BasisVAE consists of the encoder that extracts the features of data and estimates the coefficients for linear combination of the latent bases, and the decoder that reconstructs the data with the combined latent bases. In this method, a single latent basis is subject to change in a single generative factor, and relatively invariant to the changes in other factors. It maintains the performance while relaxing the constraint for disentanglement on a basis, as we no longer need to decompose latent space on a standard basis. Experiments on the well-known benchmark datasets of MNIST, 3DFaces and CelebA demonstrate the efficacy of the proposed method, compared to other state-of-the-art methods. The proposed model not only defines the latent space to be separated by the generative factors, but also shows the better quality of the generated and reconstructed images. The disentangled representation is verified with the generated images and the simple classifier trained on the output of the encoder.
[updated rating due to supervision of $c_i$, which was not made clear enough and would require other baseline models]
This paper proposes a modification of the usual parameterization of the encoder in VAEs, to more allow representing an embedding $z$ through an explicit basis $M_B$, which will be pushed to be orthogonal (and hence could correspond to a fully factorised disentangled representation). It is however possible for different samples $x$ to use different dimensions in the basis if that is beneficial (i.e. x is mapped to $z = f(x) \cdot M_B$, where f(x) = (c_1, ... , c_n) which sums to 1.). This stretches the usual definition of what a “disentangled representation” means, as this disentanglement is usually assumed to be globally consistent, but this is a fair extension.
They show that this formulation can be expressed as a different ELBO which can be maximized as for usual VAEs.
I found this paper interesting, but I have one clarification that may modify my assessment quite strongly (hence I am tentatively putting it on the accept side). Some implementation details seem missing as well. Otherwise the presentation is fair, there are several results on different datasets which demonstrate the model's behaviour appropriately.
1.	The main question I have, which may be rather trivial, is “are the c_i supervised in any way?”. When I first read the paper, and looking at the losses in equations 9-11, I thought that this wasn’t the case (also considering this paper is about unsupervised representation learning), but some sentences and figures make this quite unclear:
	a.	In Section 3.2, you say “We train the encoder so that c_i = 1 and c_j = 0 if the input data has i-feature and no j-feature”. Do you?
	b.	How are the features in Figure 6 attached to each b_i? I.e. how was “5_o_clock_shadow” attached to that particular image at the top-left?
	If the c_i are supervised, this paper is about a completely different type of generative modeling than what it compares against (it would be more comparable to VQ-VAE or other nearest-neighbor conditional density models).
2.	There is not enough details about the architecture, hyperparameter and baselines in the current version of the paper.
	a.	What n_x (i.e. dimensionality of the basis) do you use? How does this affect the results?
	b.	How exactly are f(x), \Sigma_f(x) parametrized? They mention the architecture of the “encoder” in Section 4.1, but this could be much clearer.
	c.	How do you train M_B? I assume they are just a fixed set of embeddings that are back-propagated through?
	d.	What are the details about the architecture of the baselines, and their hyperparameters? E.g. what is the beta you used for Beta-VAE?
3.	The reconstructions seem only partially related to their target inputs (e.g. see Figure 4). This seems to indicate that instead of really reconstructing x, the model chooses to reconstruct “a close-by related \tilde{x}”, or even perhaps a b_i. This would make it behave closer to VQ-VAE, which explicitly does that. How related are reconstructions/samples to the b_i?
4.	Could you show the distribution of c_i that the model learns, and how much they vary for several example images?  How “peaky” is this distribution for a given image (this feeds into to the previous question as well)? The promise of the proposed model is that different images pick and choose different combinations of b_i, which hopefully one should see reflected in the distributions of c_i per sample, across clusters, or across the whole dataset.
5.	What happens when L_B is removed? I.e. what is the effect of removing the constraint on M_B being a basis, and instead allow it to be anything? This seems to make it closer to a continuous approximation to VQ-VAE?
6.	Is Equation 10 correct? Should the KL use N(f(x) \cdot M_B, \Sigma_f(x)), as in equation 9 above?
7.	Similarly, in Section 4.2.3, did you mean “c_i = 1 and c_j = 0 for i != j”?
If the model happens to be fully unsupervised, I think that these results are quite interesting, and provide a good modification to the usual VAE framework, I find that having access to the M_B basis explicitly could be very valuable.
There is still an interesting philosophical discussion to be had about when one would like to obtain a “global basis” for the latent space (i.e. Figure 3 (b)), or when one would prefer more local ones. I can see clear advantages for a non-local basis, in terms of generalisation and compositionality, which your choice (i.e. Figure 3 (c) ) would prohibit.
References:
[1] VQ-VAE: Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu, “Neural Discrete Representation Learning”, https://arxiv.org/abs/1711.00937