Isolating Sources of Disentanglement in VAEs
We decompose the evidence lower bound to show the existence of a term measuring the total correlation between latent variables. We use this to motivate the β-TCVAE (Total Correlation Variational Autoencoder) algorithm, a refinement and plug-in replacement of the β-VAE for learning disentangled representations, requiring no additional hyperparameters during training. We further propose a principled classifier-free measure of disentanglement called the mutual information gap (MIG). We perform extensive quantitative and qualitative experiments, in both restricted and non-restricted settings, and show a strong relation between total correlation and disentanglement, when the model is trained using our framework.
This paper studies the question of why disentangled representations emerge from VAE training, and through the insights derived in this analysis, is able to propose a simple method to obtain even more disentangling in the representations. The fact that beta does not need to be carefully tuned seems like a major advantage. The proposed ideas seem quite insightful/useful, the writing is clear, and the empirical evaluation is thorough and convincing. The authors have been able to shed some light on the murky topic of disentangling, and I believe these results will be of broader interest to the ML community.   All of the issues/improvements I initially wanted to bring up, I found had already been addressed after reading through the paper a few more times.    Minor Comments:  The authors mention there being confusion about the relationship between disentanglement and mutual information between latent codes and the observed data.  Have you considered studying how beta affects this mutual information?  One major problem with VAEs is the posterior collapse phenomenon, which it seems like the beta-TCVAE would only exacerbate, correct?  Perhaps the beta-TCVAE can be combined with methods that increase the mutual information between latent codes and the observed data to avoid posterior collapse, thus ensuring a representation that is both highly disentangled and controllable.  Response to Author Feedback:  I congratulate the authors on a nice paper and recommend they include the reference to "Controllable Semantic Image Inpainting." in their discussion section.  I also recommending including one sentence about the posterior collapse phenomenon somewhere in the text.