SLICED-WASSERSTEIN AUTO-ENCODERS
In this paper we use the geometric properties of the optimal transport (OT) problem and the Wasserstein distances to define a prior distribution for the latent space of an auto-encoder. We introduce Sliced-Wasserstein Auto-Encoders (SWAE), that enable one to shape the distribution of the latent space into any samplable probability distribution without the need for training an adversarial network or having a likelihood function specified. In short, we regularize the auto-encoder loss with the sliced-Wasserstein distance between the distribution of the encoded training samples and a samplable prior distribution. We show that the proposed formulation has an efficient numerical solution that provides similar capabilities to Wasserstein Auto-Encoders (WAE) and Variational Auto-Encoders (VAE), while benefiting from an embarrassingly simple implementation. We provide extensive error analysis for our algorithm, and show its merits on three benchmark datasets.
Learning such generative models boils down to minimizing a dissimilarity measure between the data distribution and the output distribution of the generative model. To this end, and following the work of  and , we approach the problem of generative modeling from the optimal transport point of view. The optimal transport problem Villani (2008); Kolouri et al. (2017) provides a way to measure the distances between probability distributions by transporting (i.e., morphing) one distribution into another. Moreover, and as opposed to the common information theoretic dissimilarity measures (e.g., f -divergences), the p-Wasserstein dissimilarity measures that arise from the optimal transport problem: 1) are true distances, and 2) metrize a weak convergence of probability measures (at least on compact spaces). Wasserstein distances have recently attracted a lot of interest in the learning community Frogner et al. (2015); Gulrajani et al. (2017); ; ; Kolouri et al. (2017) due to their exquisite geometric characteristics Santambrogio (2015). See the supplementary material for an intuitive example showing the benefit of the Wasserstein distance over commonly used f -divergences.
In this paper, we introduce a new type of auto-encoders for generative modeling (Algorithm 1), which we call Sliced-Wasserstein auto-encoders (SWAE), that minimize the sliced-Wasserstein distance between the distribution of the encoded samples and a samplable prior distribution. Our work is most closely related to the recent work by  and more specifically the follow-up work by . However, our approach avoids the need to perform adversarial training in the encoding space and is not restricted to closed-form distributions, while still benefiting from a Wasserstein-like distance measure in the latent space. Calculating the Wasserstein distance can be computationally expensive, but our approach permits a simple numerical solution to the problem. Finally, we note that there has been several concurrent papers, including the work by Deshpande et al. (2018Deshpande et al. ( ) andŞimşekli et al. (2018, that also looked into the application of sliced-Wasserstein distance in generative modeling. Regardless of the concurrent nature of these papers, our work remains novel and is distinguished from these methods. Deshpande et al. (2018) use the sliced-Wasserstein distance to match the distributions of high-dimensional reconstructed images, which require large number of slices, O(10 4 ), while in our method and due to the distribution matching in the latent space we only need O(10) slices. We also note that Deshpande et al. (2018) proposed to learn discriminative slices to mitigate the need for a very large number of random projections that is in essence similar to the adversarial training used in GANs, which contradicts with our goal of not using adversarial training. Simşekli et al. (2018), on the other hand, take an interesting but different approach of parameter-free generative modeling via sliced-Wasserstein flows.
The authors propose a new autoencoding algorithm for the unsupervised generative modeling which they call Sliced Wasserstein Autoencoders (SWAE). SWAE minimizes a reconstruction cost (measured with respect to the non-negative cost function c(x,x') defined for pairs of input images x, x'), regularized by a penalty measuring a discrepancy between the prior distribution over the latent space qz and the push-forward pz of the unknown data distribution through the deterministic encoder. The authors present an extensive theoretical argument supporting the choice of this objective and a number of empirical results performed on MNIST, LSUN bedrooms, and Celeba. 
Even though this paper raises several interesting questions, I have several major issues with it:
****
**** 1. Claim around Equation 3 is not proved.
****
All the sections before 2.3 are providing a rather detailed theoretical argument meant to support the choice of the SWAE objective appearing in Eq. 14 of Section 2.3. Here I wand to point out to a mathematical inaccuracy in the authors' discussions, which may render the whole argument questionable. In short, the authors claim around Eq. 3 that "Eq. 3 is equivalent to Theorem 1 of [1] for deterministic encoder-decoder pairs" and don't provide any proofs for this nontrivial fact. 
The following is based on some quick derivations I did while reviewing. 
Recall that in the current paper Px is the data distribution, Py is the push-forward of Px through the superposition of the encoder \phi and decoder \psi (in other words Py is a distribution of \psi(\phi(X)) when X is distributed according Px). The authors state that:
   \inf_{\phi, \psi} Wc(Px, Py)
    is equivalent to
   (* ) \inf_{\phi, \psi} E_{X \sim Px}[ c(X, \psi(\phi(X))) ].
In other words, the authors state that using Theorem 1 of [1] they are able to show that minimizing a c-optimal transport distance between Px and Py (which is parametrized by \psi and \phi) is *equivalent* to an unconstrained optimization problem appearing on the r.h.s. of Equation 3. 
Now, the Theorem 1 of [1] referenced by authors states that if Pz is any prior distribution over the latent space and \psi * Pz is its push-forward through the deterministic decoder \psi, then the optimal transport between Px and the resulting latent variable model \psi * Pz can be equivalently written as:
   (**) Wc(Px, \psi * Pz) = \inf_{f such that f * Px = Pz} E_{X \sim Px}[ c(X, \psi(f(X))) ].
Importantly, note how the right hand side of (**) contains a constrained optimization over an auxiliary (encoder) function f, which does not appear at all in the left hand side. If the authors were to apply (**) directly, they would arrive at the following statement:
   \inf_{\phi, \psi} Wc(Px, Py)
    is equivalent to
   (*** ) \inf_{\phi, \psi} \inf_{f such that f * Px = \phi * Px} E_{X \sim Px}[ c(X, \psi(f(X))) ].
Finally, comparing (*) stated by the authors and (***) obtained above, we see that (*) is obtained by selecting one particular function f = \phi from the set {f such that f * Px = \phi * Px}. Meanwhile, this set in general may contain multiple other functions f and as a result this only shows that (*) >= (***) (as we replace \inf_f with one particular choice of f). However, in this case, I think it is indeed possible to show that (*) = (***). Imagine (***) has a global minimum at (\psi_0,\phi_0, f_0), that is the global optimum of (***) equals E_{X \sim Px}[ c(X, \psi_0(f_0(X))) ]. The same value can be achieved by (*) by setting \phi = f_0. QED. 
Once again, these are my preliminary derivations and they need to be checked. But it looks like the claim of the authors is indeed true. 
****
**** 2. Empirical evidence is not convincing. ****
****
The main topic of the paper is the unsupervised generative modeling, and the authors claim certain improvements in this field compared to the previous literature. Even though there are no ultimate evaluation metrics available in the field, recently the researchers started supporting their methods with several metrics, including FID scores. By now for most of the widely used datasets the state of art FID scores are well known. In all the experiments the authors provide pictures and interpolations (last row of Fig. 3, Fig. 5) without numbers. I would say nowadays presenting pictures is not enough (being too subjective) and at least some objective numbers (preferably FID) capturing the quality of generated samples should be reported. The authors go into detailed measurements of discrepancy between the aggregate posterior pz and the prior qz, but it is not clear how this affects the actual sample generation. Finally, it is not clear why the authors compare only to WAE-GAN and did not consider WAE-MMD, which is free of adversarial training (in contrast to WAE-GAN) and thus has a stable training and does not involve extra computations of updating the discriminator (as noted by authors on page 10).
[1] Bousquet et al., 2017.