LEARNING EMBEDDINGS INTO ENTROPIC WASSERSTEIN SPACES
Euclidean embeddings of data are fundamentally limited in their ability to capture latent semantic structures, which need not conform to Euclidean spatial assumptions. Here we consider an alternative, which embeds data as discrete probability distributions in a Wasserstein space, endowed with an optimal transport metric. Wasserstein spaces are much larger and more flexible than Euclidean spaces, in that they can successfully embed a wider variety of metric structures. We exploit this flexibility by learning an embedding that captures semantic information in the Wasserstein distance between embedded distributions. We examine empirically the representational capacity of our learned Wasserstein embeddings, showing that they can embed a wide variety of metric structures with smaller distortion than an equivalent Euclidean embedding. We also investigate an application to word embedding, demonstrating a unique advantage of Wasserstein embeddings: We can visualize the high-dimensional embedding directly, since it is a probability distribution on a low-dimensional space. This obviates the need for dimensionality reduction techniques like t-SNE for visualization.
The paper ‘Learning Discrete Wasserstein Embeddings' describes a new embedding method that,
contrary to usual embedding approaches, does not try to embed (complex, structured) data into an 
Hilbertian space where Euclidean distance is used, but rather to the space of probability measures
endowed with the Wasserstein distance. As such, data are embed on an empirical 
distribution supported by Diracs, which locations can be determined by a map that is learnt from data.
Interestingly, authors note a 'potential universality' for W_p(R^3) (from a result of Andoni et al., 2015), 
suggesting that having Diracs in R^3 could embed potentially any kind of metric on symbolic data. 
Experimental validations are presented on graph and word embedding, and a discussion on visualization of 
the embedding is also proposed (since the Diracs are located in a low dimensional space).   
All in all the paper is very clear and interesting. The idea of embedding in a Wasserstein space is 
original (up to my knowledge) and well described. I definitely believe that this work should be presented
at ICLR. I have a couple of questions and remarks for the authors:
 - It is noted in section 3.2 that both Diracs location and associated weights could be optimized. Yet the authors 
   chose to only optimize locations. Why not only optimizing the weights (as in an Eulerian view of probability
   distributions) ? The sentence involving works of Brancolini and Claici 2018 is not clear to me. Why weighting 
   does not improve asymptotically the approximation quality ? 
 - Introducing the entropic regularization is mainly done for being able to differentiate the Wasserstein 
   distance. However, few is said on the embeddability of metrics in W^\lambda_p(R). Is using an entropic 
   version of W moderating the capacity of embedding ? At least experimentally, a discussion could be made  
   on the choice of the regularization parameter, at least in section 4.1. In eq. (9), it seems that it is not 
   the regularized version of W. ? 
 - I assume that the mapping is hard to invert, but did the authors tried to experiment reconstructing an object 
   of interest by following a geodesic in the Wasserstein space ?  
 - It seems to me that authors never give generalization results. What is the performance of the metric approximation 
   when tested on unseen graphs or words ? This point should be clarified in the experiment.