Optimal Transport offers an alternative to maximum likelihood for learning generative autoencoding models. We show how this principle dictates the minimization of the Wasserstein distance between the encoder aggregated posterior and the prior, plus a reconstruction error. We prove that in the non-parametric limit the autoencoder generates the data distribution if and only if the two distributions match exactly, and that the optimum can be obtained by deterministic autoencoders. We then introduce the Sinkhorn AutoEncoder (SAE), which casts the problem into Optimal Transport on the latent space. The resulting Wasserstein distance is minimized by backpropagating through the Sinkhorn algorithm. SAE models the aggregated posterior as an implicit distribution and therefore does not need a reparameterization trick for gradients estimation. Moreover, it requires virtually no adaptation to different prior distributions. We demonstrate its flexibility by considering models with hyperspherical and Dirichlet priors, as well as a simple case of probabilistic programming. SAE matches or outperforms other autoencoding models in visual quality and FID scores.
The paper introduces a new cost function for training Wasserstein Autoencoders that combines reconstruction error with Sinkhorn distance on the latent space. Authors provide nice theoretical motivation, yet empirical results seem incremental and do not fully support the effectiveness of this approach.
Pros:
- Theorem 3.1 (although trivial) provides motivation for optimizing Wasserstein distance in the latent space in WAEs.
- Theorem 3.2 shows sufficiency of optimization over deterministic encoders in WAEs.
- The proposed SAE virtually does not favor any prior and can preserve some aspects of geometry of the original space. 
Cons:
- It is unclear why Sinkhorn algorithm would provide better estimate of Wasserstein distance than e.g. adversarial WGANGP (which would be a variant of GAN-WAE). Sinkhorn convergence is discussed only in terms of sample size and  smoothing regularizer, not in the context of batch training. 
- Quantitative results are on par or marginally better than other methods, they also lack some comparisons (see details below).
- There is no comparison to relevant models outside VAE scope, e.g. ALI [4]. 
The novelty of this paper is combining WAEs with Sinkhorn algorithm. Overall, it has potential, but the proposed method would probably require clearer evaluation. 
Detailed issues:
- Notation for posterior seems somewhat inconsistent and misleading, namely push-forward G#P_Z = P_G, while Q#P_X = Q_Z.
- It is unclear why MMD or GAN losses on WAS's latent space are referred to as heuristics, each of these constitutes a divergence in the same way as the proposed Sinkhorn distance.
- FID scores for MNIST are incomparable due to the use of own network; using LeNet has been proposed [3].
- It is unclear what ‘Empirical lower bounds’ for MMD mentioned in Table 1. caption mean, as unbiased MMD estimator (e.g. [2]) is available. On the other hand, FID is known to be biased [3], so test-set FID should be provided for comparison.
- Table 2. lacks comparison of SAE with normal prior even though a) authors note that MMDs are incomparable with different priors, b) SAEs is claimed to be prior-agnostic, c) in such setting MMD-WAE might be advantageous [1]. Again, no test-set FID scores.
- Samples in Figure 2 too small.
- MMD lacks citation (e.g. [2]).
Typos:
p.6 line 3 construcetion -> construction
p.6 line 30 Hypersherical -> Hyperspherical 
P.8 line 1 this a sign -> this is a sign
[1] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein Auto-Encoders. ICLR 2018.
[2] Arthur Gretton, Karsten M. Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alex J. Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13, 2012a.
[3] Mikołaj Bińkowski, Dougal J. Sutherland, Michael Arbel, Arthur Gretton. Demystifying MMD GANs. ICLR 2018.
[4] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky and Aaron Courville. Adversarially Learned Inference. ICLR 2017