Information Constraints on Auto-Encoding Variational Bayes
Parameterizing the approximate posterior of a generative model with neural networks has become a common theme in recent machine learning research. While providing appealing flexibility, this approach makes it difficult to impose or assess structural constraints such as conditional independence. We propose a framework for learning representations that relies on auto-encoding variational Bayes, in which the search space is constrained via kernel-based measures of independence. In particular, our method employs the d-variable Hilbert-Schmidt Independence Criterion (dHSIC) to enforce independence between the latent representations and arbitrary nuisance factors. We show how this method can be applied to a range of problems, including problems that involve learning invariant and conditionally independent representations. We also present a full-fledged application to singlecell RNA sequencing (scRNA-seq). In this setting the biological signal is mixed in complex ways with sequencing errors and sampling effects. We show that our method outperforms the state-of-the-art approach in this domain.
In this work, the authors propose an approach to enforce independence among components of latent representations learned with AEVB. This amounts to adding a penalty term based on the Hilbert-Schmidt Independence Criterion to the ELBO. A set of experiments suggest the approach learns independent components.  The problem is well-motivated, and theoretic justification is given for why the approach could be expected to work. The experiments are also reasonable, but the lack of interpretation of the results on the real datasets makes it difficult to tell whether the results are in some sense “good.” There is also little theoretical comparison between the proposed approach and similar (cited) work.  === Major comments  As mentioned above, the lack of explanation of the experimental results makes it difficult to interpret them. For example, in Table 1, the *worst* results for the lighting group and direction are in bold. Presumably, that is actually good because it means the latent representations are ignoring the nuisance (lighting) information. However, a more clear explanation would make this point more obvious. Similarly, it is not obvious what the evaluation metrics used on the sc-seq data really show.  The main difference compared to \beta-TCVAE and VFAE seems to be the choice of mutual information measure. Thus, it would be helpful to more explicitly compare dHSIC and the measures used in the other approaches. Otherwise, it is not clear why we would expect the proposed approach to outperform those.  === Minor comments  At the end of Section 3.1, the authors propound that “the independence properties… will often not be respected by the approximate posterior.” While I agree with this, a reference or two which either prove or demonstrate this empirically would be helpful.  Figure 2 should also show results for an unconstrained VAE as a baseline.  For reference, Table 1 should include unconstrained VAE results. This would make more clear what is “lost” (in terms of accuracy) by encouraging independence among the latent features.   === Typos, etc.  “equation 5” -> “Equation 5”  The “superscript” notation (presumably indicating a single dimension/variable in the respective space) is not defined.  The fonts on Figure 2 should be increased. Also, it is not understandable when printed in black and white. Different marker types could fix that.  “has be shown” -> “has been shown”  MMD is not defined.  “Dirichlet kernel” in Proposition 2. This seems more like the definition of the Dirac delta function.  The references are inconsistently formatted. Also, some of the names are misspelled, like “Schlkopf”.  --- After author feedback  The authors have addressed my concerns.