Bayesian Manifold Learning: The Locally Linear Latent Variable Model
We introduce the Locally Linear Latent Variable Model (LL-LVM), a probabilistic model for non-linear manifold discovery that describes a joint distribution over observations, their manifold coordinates and locally linear maps conditioned on a set of neighbourhood relationships. The model allows straightforward variational optimisation of the posterior distribution on coordinates and locally linear maps from the latent space to the observation space given the data. Thus, the LL-LVM encapsulates the local-geometry preserving intuitions that underlie non-probabilistic methods such as locally linear embedding (LLE). Its probabilistic semantics make it easy to evaluate the quality of hypothesised neighbourhood relationships, select the intrinsic dimensionality of the manifold, construct out-of-sample extensions and to combine the manifold model with additional probabilistic models that capture the structure of coordinates within the manifold.
The paper introduces a model which is probabilistic for non linear manifold discovery. It is based on a generative model with missing variables and required a variational EM implementation which is standard but nevertheless technical to derive in this specific context.As regards the bibliography, I can see at least two relevant references missing: [1] A. Deleforge, F. Forbes and R. Horaud, High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variables. Statistics and Computing.[2] A. Deleforge, F. Forbes, S. Ba and R. Horaud, Hyper-spectral image analysis with Partially-Latent Regression and spatial Markov dependencies. IEEE journal of selected topics in signal processing 2015.Note that [2] is an extension of [1] with a Markov neighborhood and a VEM approach to high-to-low dimensional mapping. In terms of results, the authors seem to stop their EM algorithm after a rather small (30-40) number of iterations without any mention of any more formal convergence or stoppping criterion. Why is it so? This looks a little suspicious. Do the results degrade when more iterations are done? In figure 3 B, I cannot see the mentioned graph structure? Comparison with GP-LVM: this paragraph is missing a clear conclusion at the end. I'd suggest the authors explain better what they conclude from this comparison. Fig 5 F: is the difference between the first 3 methods actually significant? Fig 5-A: The authors mention multiple initializations of EM but what is their conclusion wrt the sensitivity of their algorithm to initialization? Fig 6: I would need more information as why the authors think that the result of their method, although better than the others, is satisfying. It seems to me, at least visually, that the recovered mapping is quite far from the ground truth one.In their conclusion, the authors mention that their model permits evaluation of hypothetical neighborhood but I do not see where in the paper this is illustrated.