Generative modelling is often cast as minimizing a similarity measure between a data distribution and a model distribution. Recently, a popular choice for the similarity measure has been the Wasserstein metric, which can be expressed in the Kantorovich duality formulation as the optimum difference of the expected values of a potential function under the real data distribution and the model hypothesis. In practice, the potential is approximated with a neural network and is called the discriminator. Duality constraints on the function class of the discriminator are enforced approximately, and the expectations are estimated from samples. This gives at least three sources of errors: the approximated discriminator and constraints, the estimation of the expectation value, and the optimization required to find the optimal potential. In this work, we study how well the methods, that are used in generative adversarial networks to approximate the Wasserstein metric, perform. We consider, in particular, the c-transform formulation, which eliminates the need to enforce the constraints explicitly. We demonstrate that the c-transform allows for a more accurate estimation of the true Wasserstein metric from samples, but surprisingly, does not perform the best in the generative setting.
values. The discriminators are functions from the sample space to the real line, that have different interpretations in different GAN variations. The main technical issue is that the discriminators have to satisfy specific conditions, such as being 1-Lipschitz in the 1-Wasserstein case. In the original paper, this was enforced by clipping the weights of the discriminator to lie inside some small box, which, however, proved to be inefficient. The Gradient penalty WGAN (WGAN-GP) (Gulrajani et al., 2017) was more successful at this, by enforcing the constraint through a gradient norm penalization. Another notable improvement was given by the consistency term WGAN (CT-WGAN) (Wei et al., 2018), which penalizes diverging from 1-Lipschitzness directly. Other derivative work of the WGAN include different OT inspired similarity measures between distributions, such as the sliced Wasserstein distance (Deshpande et al., 2018), the Sinkhorn divergence (Genevay et al., 2018) and the Wasserstein divergence (Wu et al., 2018). Another line of work studies how to incorporate more general ground cost functions than the l 2 -metric (Adler & Lunz, 2018;Mallasto et al., 2019;Dukler et al., 2019).
Recent works have studied the convergence of estimates of the Wasserstein distance between two probability distributions, both in the case of continuous (Klein et al., 2017) and finite (Sommerfeld & Munk, 2018;Sommerfeld, 2017) sample spaces. The decay rate of the approximation error of estimating the true distance with minibatches of size N is of order O(N −1/d ) for the Wasserstein distances, where d is the dimension of the sample space (Weed & Bach, 2017). Entropic regularized optimal transport has more favorable sample complexity of order O(1/ √ N ) for suitable choices of regularization strength (see Genevay et al. 2019 andalso Feydy et al. 2018;Mena & Weed 2019). For this reason, entropic relaxation of the 1-Wasserstein distance is also considered in this work.
Contribution. In this work, we study the efficiency and stability of computing the Wasserstein metric through its dual formulation under different schemes presented in the WGAN literature. We present a detailed discussion on how the different approaches arise and differ from each other qualitatively, and finally measure the differences in performance quantitatively. This is done by quantifying how much the estimates differ from accurately computed ground truth values between subsets of commonly used datasets. Finally, we measure how well the distance is approximated during the training of a generative model. This results in a surprising observation; the method best approximating the Wasserstein distance does not produce the best looking images in the generative setting.
[Summary]
This paper provides an empirical evaluation of commonly used discriminator training strategies in estimating the Wasserstein distance between distributions. The paper finds that methods motivated from optimal transport theory, e.g. c-transform and (c,\eps)-transform, perform better in evaluating the Wasserstein distance than methods commonly used in WGAN practice such as weight clipping and gradient penalty. However, when deployed in WGANs as the discriminator training strategy, these methods do not generate images as high-quality as the gradient penalty method. 
[Pros]
The question considered in this paper, i.e. how well does various discriminator strategies (as proxies of the infeasible all 1-Lipschitz discriminator) perform for *evaluating* the Wasserstein distributions, is important for strengthening our understanding of generative models. The main result that methods that are better at computing W may not be better at generating images is interesting, and agrees with the theoretical insight (e.g. Arora et al. 2017) that *non-parametric* minimization of W may not be a good explanation for the generative power of WGANs.
[Cons]
I have concerns about the specific setting of “mini-batch distance” considered in this paper, in that whether it is really a sensible task (and can say anything about computing W) given the curse of dimensionality of estimating W from samples. The paper did not really discuss this issue, and from my own thoughts I don’t think the task avoids this issue. 
From my understanding, the “Approximation” experiment does the following:
(1) Set (\mu, \nu) to be a random split of a dataset and consider them to be the populations. Let’s let f_\star denote the (ground truth) optimal discriminator between (\mu, \nu).
(2) Train discriminators (f_wc, f_gp, f_c, f_ceps) (using the different algorithms) from training batches from (\mu, \nu).
(3) Evaluate <f, \mu’_l - \nu’_l> where (\mu’_l, \nu’_l) are fresh test batches from (\mu, \nu) and f is one of the above trained discriminators.
In comparison, the “ground truth” computes W(\mu’_l, \nu’_l) = <f_l, \mu’_l, \nu’_l> from the POT package (though not necessarily through explicitly computing f_l.)
The issue with this is that we expect the method to perform well if f ~= f_l, which can be achievable if all the f_l’s are similar (and hopefully they’re all approximately equal to f_\star.) However I don’t think this is true -- as (\mu’_l, \nu’_l) are samples, and because of the curse of dimensionality, we should expect the f_l’s to be quite different from each other. (Otherwise if they’re really just ~= f_\star, then we can use standard concentration to show the W(\mu’_l, \nu’_l) ~= W(\mu, \nu), which we know is not true from curse of dim.)
Given this, I don’t think the task of comparing <f, \mu’_l, \nu’_l> with the POT results really says anything about their power in computing W. I would be glad though to hear back from the authors to see if my understanding is accurate, and adjust my evaluation from there.