We propose a metric-learning framework for computing distance-preserving maps that generate low-dimensional embeddings for a certain class of manifolds. We employ Siamese networks to solve the problem of least squares multidimensional scaling for generating mappings that preserve geodesic distances on the manifold. In contrast to previous parametric manifold learning methods we show a substantial reduction in training effort enabled by the computation of geodesic distances in a farthest point sampling strategy. Additionally, the use of a network to model the distance-preserving map reduces the complexity of the multidimensional scaling problem and leads to an improved non-local generalization of the manifold compared to analogous non-parametric counterparts. We demonstrate our claims on point-cloud data and on image manifolds and show a numerical analysis of our technique to facilitate a greater understanding of the representational power of neural networks in modeling manifold data.
The authors argue that the spectral dimensionality reduction techniques are too slow, due to the complexity of computing the eigenvalue decomposition, and that they are not suitable for out-of-sample extension. They also note the limitation of neural networks, which require huge amounts of data to properly learn the data structure. The authors therefore propose to first sub-sample the data and afterwards to learn an MDS-like cost function directly with a neural network, resulting in a parametric framework.
The paper should be checked for grammatical errors, such as e.g. consistent use of (no) hyphen in low-dimensional (or low dimensional).
The abbreviations should be written out on the first use, e.g. MLP, MDS, LLE, etc.
In the introduction the authors claim that the complexity of parametric techniques does not depend on the number of data points, or that moving to parametric techniques would reduce memory and computational complexities. This is in general not true. Even if the number of parameters is small, learning them might require complex computations on the whole data set. On the other hand, even if the number of parameters is equal to the number of data points, the computations could be trivial, thus resulting in a complexity of O(N).
In section 2.1, the authors claim "Spectral techniques are non-parametric in nature"; this is wrong again. E.g. PCA can be formulated as MDS (thus spectral), but can be seen as a parametric mapping which can be used to project new words.
In section 2.2, it says "observation that the double centering...". Can you provide a citation for this?
In section 3, the authors propose they technique, which should be faster and require less data than the previous methods, but to support their claim, they do not perform an analysis of computational complexity. It is not quite clear from the text what the resulting complexity would be. With N as number of data points and M as number of landmarks, from the description on page 4 it seems the complexity would be O(N + M^2), but the steps 1 and 2 on page 5 suggest it would be O(N^2 + M^2). Unfortunately, it is also not clear what the complexity of previous techniques, e.g DrLim, is.
Figure 3, contrary to text, does not provide a visualisation to the sampling mechanism.
In the experiments section, can you provide a citation for ADAM and explain how the parameters were selected? Also, it is not meaningful to measure the quality of a visualisation via the MDS fit. There are more useful approaches to this task, such as the quality framework [*].
In figure 4a, x-axis should be "number of landmarks".
It is not clear why the equation 6 holds. Citation?
It is also not clear how exactly the equation 7 is evaluated. It says "By varying the number of layers and the number of nodes...", but the nodes and layer are not a part of the equation.
The notation for equation 8 is not explained.
Figure 6a shows visualisations by different techniques and is evaluated "by looking at it". Again, use [*].
[*] Lee, John Aldo ; Verleysen, Michel. Scale-independent quality criteria for dimensionality reduction. In: Pattern Recognition Letters, Vol. 31, no. 14, p. 2248-2257 (2010). doi:10.1016/j.patrec.2010.04.013.