Probabilistic Curve Learning: Coulomb Repulsion and the Electrostatic Gaussian Process
Learning of low dimensional structure in multidimensional data is a canonical problem in machine learning. One common approach is to suppose that the observed data are close to a lower-dimensional smooth manifold. There are a rich variety of manifold learning methods available, which allow mapping of data points to the manifold. However, there is a clear lack of probabilistic methods that allow learning of the manifold along with the generative distribution of the observed data. The best attempt is the Gaussian process latent variable model (GP-LVM), but identifiability issues lead to poor performance. We solve these issues by proposing a novel Coulomb repulsive process (Corp) for locations of points on the manifold, inspired by physical models of electrostatic interactions among particles. Combining this process with a GP prior for the mapping function yields a novel electrostatic GP (electroGP) process. Focusing on the simple case of a one-dimensional manifold, we develop efficient inference algorithms, and illustrate substantially improved performance in a variety of experiments including filling in missing frames in video.
This paper proposes a new method to learn a mapping function from a d-dimensional data point, y, to one-dimensional embedding, x. The paper extends the standard GP to a hierarchical model by incorporating what they call Coulomb repulsive process (Corp). It first generates samples of the one-dimensional x and then generates observable d-dim data points through a GP, whose covariance is constrained to be diagonal. Intuitively, Corp is used to ensure reasonable inter-point distance in the embedding space through the repulsive force in [0,1] to make the distance of 0.5 is the least stressful state. The equi-distance prior in [0,1] looks like a natural choice when the original distribution for y forms a compact single continuous manifold in the original space. To find the embeddings, they propose an alternating algorithm, where GP parameters and the embeddings are solved alternatingly. To the best of my knowledge, the use of Corp in the context of GP-LVM is new. The proposed approach looks interesting enough to accept the paper. I personally think that the paper can be improved by describing theoretical aspects of the model more carefully. It looks obvious to me that the model still has major limitations such as the strong dependency on the initial guess (x_0). I feel that the authors are overselling the model by making too strong statements on the practical utility of the model. For example, in practice, we all know that careful pre-process is required to make "your favorite distance-preserving manifold" work. What if the distribution looks multi-modal? What if x_0 misses to capture the major features of the original distribution? Your claim in Sec. 5 is scientifically sound? I know recently more and more reviewers tend to demand extensive experimental comparison. However, for papers like this, the most important thing is to build a common understanding with the audience on the major features of the model and wide applicability of the model. I hope the camera-ready version gets more understandable and sophisticated by addressing it. Minor comments. - Do not mix between bold and non-bold notation to represent vectors. Use boldface for e.g. \mu in Introduction. - Introduction of Corp in Sec.2 can be improved. This is basically to introduce the repulsive force. Don't make it overly mathematical. - If you are to use r=1, you don't have to show r=20. You can show illustrative figures instead. - Dependency on x_0 should be discussed in depth. If you cannot, do not overclaim the practical utility. - Also, describe practical limitations in addition to practical advantages. - Sec. 5 sounds like a frivolous sales talk. This is a good paper as is. You don't have to worry about the "more experiment!" criticism. I'd like you to make more sophisticated academic comments.   - Redraw Fig.2. Visibility is extremely bad.