Variational inference (VI) methods and especially variational autoencoders (VAEs) specify scalable generative models that enjoy an intuitive connection to manifold learning -with many default priors the posterior/likelihood pair q(z|x)/p(x|z) can be viewed as an approximate homeomorphism (and its inverse) between the data manifold and a latent Euclidean space. However, these approximations are well-documented to become degenerate in training. Unless the subjective prior is carefully chosen, the topologies of the prior and data distributions often will not match. Conversely, diffusion maps (DM) automatically infer the data topology and enjoy a rigorous connection to manifold learning, but do not scale easily or provide the inverse homeomorphism. In this paper, we propose a) a principled measure for recognizing the mismatch between data and latent distributions and b) a method that combines the advantages of variational inference and diffusion maps to learn a homeomorphic generative model. The measure, the locally bi-Lipschitz property, is a sufficient condition for a homeomorphism and easy to compute and interpret. The method, the variational diffusion autoencoder (VDAE), is a novel generative algorithm that first infers the topology of the data distribution, then models a diffusion random walk over the data. To achieve efficient computation in VDAEs, we use stochastic versions of both variational inference and manifold learning optimization. We prove approximation theoretic results for the dimension dependence of VDAEs, and that locally isotropic sampling in the latent space results in a random walk over the reconstructed manifold. Finally, we demonstrate our method on various real and synthetic datasets, and show that it exhibits performance superior to other generative models.
The paper proposes a new generative model for unsupervised learning, based on a diffusion random walk principle inspired by the manifold learning literature. The basic idea is to (probabilistically) map points to a latent space, perform a random walk in that space, and then map back to the original space again. Learning of the suitable maps is achieved by casting the problem in a variational inference framework.
The paper is generally well-written, and clearly states out its goals and motivation. Sections 1 - 3 in particular give a nice overview of the broader context of the paper, and its aim of borrowing ideas from manifold learning and variational autoencoders. The particular aim on using concepts from manifold learning to avoid mode collapse - corresponding to the underlying homeomorphism losing its bijectivity - is in particular intriguing.
The method itself is intuitive at a high level, although I did have some difficulty with Section 4:
- in 4.1, one begins by considering the local evidence. This requires drawing a point from U_x, which is defined to a be set. I presume this means one draws uniformly from this set?
- Eqn 3 does not apparently have the same structure as Eqn 2. In particular, the first term in (2) is a function of x, but for (3) it is not a function of x'. How does conditioning affect the ELBO?
- I was not sure how to interpret the statement that pθ(x'|x) ≈ ψ^{-1}(q(z'|x)). Do you mean the distribution is strongly concentrated around this value? Note also an extra "]" in the latter.
- Eqn 5 should presumably be an equality? Also, it was not clear what the d in |.|_d^2 means, and why one does not use ||.||^2.
- At a higher level, given that x' ~ U_x originally, why do we now draw x' ~ p(x'|x)?
- Arriving at 4.2, it was not clear what "The sampling procedure" refers to, i.e., which of the steps in 4.1 it is seeking to specify or augment. It would be useful to clearly lay out the objective function that is being optimised, and how this section fits into that.
- In 4.3, it seemed as if the discussion of the neighbourhood reconstruction error would be better placed in 4.1 itself. It appears to be a justification of the already-derived Eqn 5.
- Algorithm 2, is there a need to introduce Z_t? It is a bit confusing that, e.g., Z_1 is first written to in iteration 1 by g(Z_0, ε), and then by ψ(X_1) in the second iteration.
The authors also claim a contribution to be the identification of a principled measure to identify mismatch between latent and data distributions. This "bi-Lipschitz" property is only introduced in Sec 5.1, and the discussion is not too approachable to someone unfamiliar with the area. In particular:
- it is not clear how precisely the discussion in this section relates to the VDAE algorithm described in the previous section.
- precisely what quantity we are to compute so as to verify this condition remains elusive. The abstract and introduction made me expect that the property is practically verifiable, but it was not clear from this section whether it is so.
- the conclusion or key takeaway of this subsection was unclear. I gather that Jones et al. established the existence of a neighbourhood wherein one can define a bi-Lipschitz mapping to R^d for suitable d. But how does this relate to latent and data space mismatch?
The experiments show that the proposed method can generate meaningful samples for synthetic manifold data, as well as on the MNIST dataset. I would've preferred more discussion of the results in Sec 4.1. I also was hoping for a clearer illustration of mode-collapse problems on standard benchmarks for GANs, with comparison of results to, e.g., those of Wasserstein-GANs (beyond those in Sec 6.1) or other proposals that aim to mitigate mode collapse.
Minor comments:
- Fig 1, the text in the middle panel is hard to read in black and white.
- SpectralNet is mentioned a few times but never formally introduced.
- notationally, Section 4 is a little heavy. I would suggest considering to omit the subscript θ's in the function ψ and its inverse.
- when mentioning the "reparametrisation trick", please provide a citation.
- what is \mathbb{X} in Theorem 2?
- "completementary" -> "complementary"
- " rejection sampling Azadi et al. (2018); Turner et al. (2018)." -> " rejection sampling (Azadi et al. 2018; Turner et al. 2018)."
- some form of Conclusion would be appropriate.