In a number of practical applications that rely on dimensionality reduction, the dataset or measurement process provides valuable side information that can be incorporated when learning low-dimensional embeddings. We propose the inclusion of pairwise representation constraints into autoencoders (AEs) with the goal of promoting application-specific structure. We use synthetic results to show that only a small amount of AE representation constraints are required to substantially improve the local and global neighborhood preserving properties of the learned embeddings. To demonstrate the efficacy of our approach and to illustrate a practical application that naturally provides such representation constraints, we focus on wireless positioning using a recently proposed channel charting framework. We show that representation-constrained AEs recover the global geometry of the learned low-dimensional representations, which enables channel charting to perform approximate positioning without access to global navigation satellite systems or supervised learning methods that rely on extensive measurement campaigns.
Description:
This paper presents a variant of deep neural network autoencoders for low-dimensional embedding, where pairwise constraints are incorporated, and applies it to wireless positioning.
The four constraint types are about enforcing pairwise distance between low-dimensional points to be close to a desired value or below a maximal desired value, either as an "absolute" constraint where one point is fixed or a "relative" constraint where both points are optimized. The constraints are encoded as  nonconvex regularization terms. In addition to the constraints the method has a standard autoencoder cost function.
Authors point out that if a suitable importance weighting is done, one constraint type yields a parametric version of Sammon’s mapping.
The method is tested on four simple artificial manifolds and on a wireless positioning task.
Evaluation:
Combining autoencoders with suitable additional regularizers can be a meaningful approach. However, I find the evaluation of the proposed method very insufficient: there are no comparisons to any other dimensionality reduction methods. For example, Sammon's mapping is mentioned several times but is not compared to, and a parametric version of t-SNE is also mentioned but not compared to even though it is parametric like the authors' proposed method. I consider that to be a severe problem in a situation where numerous such methods have been proposed previously and would be applicable to the data used here.
In terms of novelty I find the method somewhat lacking: essentially it is close to simply a weighted combination of an AE cost function and a Sammon's mapping cost function when using the FRD constraints. The other types of constraints add some more novelty, however.
Detailed comments:
"Autoencoders have been shown to consistently outperform other dimensionality-reduction algorithms on real-world datasets (van der Maaten et al., 2009)": this is too old a reference, nine years old, and it does not contain numerous dimensionality reduction algorithms proposed more recently, such as any neighbor embedding based dimensionality reduction methods. Moreover, the test in van der Maate et al. 2009 was only on five data sets and in terms of a continuity measure only, too little evidence to claim consistent outperforming of other algorithms.
"van der Maaten (2009) proposes the use of AEs to learn a parametric mapping between high-dimensional datapoints and low-dimensional representations by enforcing structure obtained via Student-t stochastic neighborhood embedding (t-SNE)": this is not a correct description, van der Maaten (2009) optimizes the AE using t-SNE cost function (instead of running some separate t-SNE step to yield structural constraints as the description seems to say).
"the FRD regularizer resembles that of Sammon's mapping": actually in the general form it resembles the multidimensional scaling stress; it only becomes close to Sammon's mapping if you additionally weight each constraint by the inverse of the original distance as you suggest.
It is unclear to me where the absolute distance constraints (FAD or MAD) arise from in the synthetic experiments. You write "for FAD one of the two representations... is a constant known prior to AE learning": how can you know the desired low-dimensional output coordinate (or distance from such a coordinate) in the synthetic data case?
This reference is incorrect: "Laurens van der Maaten, Eric Postma, and Jaap Van den Herik. Dimensionality reduction: A comparative review. In Journal of Machine Learning Research, volume 10, pp. 66–71, 2009." This article has not been published in Journal of Machine Learning Research. It is only available as a technical report of Tilburg University.