We consider the question of how to assess generative adversarial networks, in particular with respect to whether or not they generalise beyond memorising the training data. We propose a simple procedure for assessing generative adversarial network performance based on a principled consideration of what the actual goal of generalisation is. Our approach involves using a test set to estimate the Wasserstein distance between the generative distribution produced by our procedure, and the underlying data distribution. We use this procedure to assess the performance of several modern generative adversarial network architectures. We find that this procedure is sensitive to the choice of ground metric on the underlying data space, and suggest a choice of ground metric that substantially improves performance. We finally suggest that attending to the ground metric used in Wasserstein generative adversarial network training may be fruitful, and provide a concrete formulation for doing so.
`The papers aims to provide a quality measure/test for GANs.   The objective is ambitious an deserve attention. As GANs are minimizing some f-divergence measure, the papers remarks that computing a  Wasserstein distance between two distributions made of a sum of Diracs is not a degenerate case and is tractable. So they propose evaluate the current approximation of a distribution learnt by a GAN by using this distance as a baseline performance (in terms of W distance and computed on a hold out dataset). 
A first remark is that the papers does not clearly develop the interest of puting things a trying to reach a treshold of performance in W distance rather than just trying to minimize the desired f-divergence. More specifically as they assess the performance in terms of W distance I would would be tempted to just minimize the given criterion. This would be very interesting to have arguments on why being better than the "Dirac estimation" in terms of W2 distance would lead to better performance for others tasks (as other f-divergences or image generation).
According to the authors the core claims are:
"1/ We suggest a formalisation of the goal of GAN training (/generative modelling more broadly) in terms of divergence minimisation. This leads to a natural, testable notion of generalisation. "
Formalization in terms of divergence minimization is not new (see O. Bousquet & all https://arxiv.org/pdf/1701.02386.pdf ) and I do not feel like this paper actually performs any "test" (in a statistical sense). In my opinion the contribution is more about exhibiting a baseline which has to be defeated for any algorithm interesting is learning the distribution in terms of W2 distance.
"2/ We use this test to evaluate the success of GAN algorithms empirically, with the Wasserstein distance as our divergence."
Here the distance does not seems so good because the performance in generation does not seems to only be related to W2 distance. Nevertheless, there is interesting observations in the paper about the sensitivity of this metric to the bluring of pictures. I would enjoyed more digging in this direction. The authors proposes to solve this issue by relying to an embedded space where the L2 distance makes more sense for pictures (DenseNet). This is of course very reasonable but I would expect anyone working on distribution over picture to work with such embeddings. Here I'm not sure if this papers opens a new way to improve the embedding making use on non labelled data. One could think about allowing the weights of the embeddings to vary while f-divergence is minimized but this is not done in the submitted work.
 "3/ We find that whether our proposed test matches our intuitive sense of GAN quality depends heavily on the ground metric used for the Wasserstein distance."
This claim is highly biased by who is giving the "intuitive sense". It would be much better evaluated thought a mechanical turk test.
 "4/ We discuss how to use these insights to improve the design of WGANs more generally."
As our understanding of the GANs dynamics are very coarse, I feel this is not a good thing to claim that "doing xxx should improve things" without actually trying it.