The Cramer Distance as a Solution to Biased Wasserstein Gradients
The Wasserstein probability metric has received much attention from the machine learning community. Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes. The value of being sensitive to this geometry has been demonstrated, among others, in ordinal regression and generative modelling, and most recently in reinforcement learning. In this paper we describe three natural properties of probability divergences that we believe reflect requirements from machine learning: sum invariance, scale sensitivity, and unbiased sample gradients. The Wasserstein metric possesses the first two properties but, unlike the Kullback-Leibler divergence, does not possess the third. We provide empirical evidence suggesting this is a serious issue in practice. Leveraging insights from probabilistic forecasting we propose an alternative to the Wasserstein metric, the Cramér distance. We show that the Cramér distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences. We give empirical results on a number of domains comparing these three divergences. To illustrate the practical relevance of the Cramér distance we design a new algorithm, the Cramér Generative Adversarial Network (GAN), and show that it has a number of desirable properties over the related Wasserstein GAN.
 The contribution of the article is related to performance criteria, and in particular to the Wasserstein/Mallows metric, which has received a good deal of attention these last few years in the machine learning literature. The paper starts with a discussion about desirable properties of a loss function and points out the fact that (plug-in) empirical versions of the gradient of this quantity are biased, which limits its interest, insofar as many learning techniques are based on (stochastic) gradient descent. In its current state, this argument looks artificial. Indeed, zero bias can be a desirable properties for an estimate but being biased does not prevent it from being accurate. In contrast, in many situations like ridge regression, incorporating bias permits to drastically reduce variance.It quite depends on the structural assumptions made. For this reason, the worst case result (Theorem 1) is not that informative in my opinion. As they are mainly concerned by the L_1 version of the Wasserstein distance, rather than focussing on the bias, the authors could consider the formulation in terms of inverse cumulative distribution functions in the 1-d setup and the fact that the empirical cdf is never invertible: even if the theoretical cdf is invertible (which naturally guarantees uniqueness of the optimal transport) the underlying mass transportation problem is not as well-posed as that related to its statistical counterpart (however, smoothing the empirical distribution may remedy this issue). The authors propose to use instead the Cramer distance, which is a very popular distance in Statistics and on which many statistical hypothesis testing procedures rely, and review its appealing properties. The comparisons between KL, Wasserstein and Cramer distances is vain in my opinion and willing to come to a general conclusion about the merits of one against the others is naive. In a nonparametric setup, it is always possible to find distributions such that certain of its properties are hidden by certain distances and highlighted by others. This is precisely why you are forced to specify the type of deviations between distributions in nonparametric hypothesis testing (a shift, a change in scale, etc.), there is no way of assessing universally that two distributions are close: optimality can only be assessed for sequences of contiguous hypotheses. The choice of the distance is part of the learning problem.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
The Cramer Distance as a Solution to Biased Wasserstein Gradients
The Wasserstein probability metric has received much attention from the machine learning community. Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes. The value of being sensitive to this geometry has been demonstrated, among others, in ordinal regression and generative modelling, and most recently in reinforcement learning. In this paper we describe three natural properties of probability divergences that we believe reflect requirements from machine learning: sum invariance, scale sensitivity, and unbiased sample gradients. The Wasserstein metric possesses the first two properties but, unlike the Kullback-Leibler divergence, does not possess the third. We provide empirical evidence suggesting this is a serious issue in practice. Leveraging insights from probabilistic forecasting we propose an alternative to the Wasserstein metric, the Cramér distance. We show that the Cramér distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences. We give empirical results on a number of domains comparing these three divergences. To illustrate the practical relevance of the Cramér distance we design a new algorithm, the Cramér Generative Adversarial Network (GAN), and show that it has a number of desirable properties over the related Wasserstein GAN.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The contribution of the article is related to performance criteria, and in particular to the Wasserstein/Mallows metric, which has received a good deal of attention these last few years in the machine learning literature. The paper starts with a discussion about desirable properties of a loss function and points out the fact that (plug-in) empirical versions of the gradient of this quantity are biased, which limits its interest, insofar as many learning techniques are based on (stochastic) gradient descent. In its current state, this argument looks artificial. Indeed, zero bias can be a desirable properties for an estimate but being biased does not prevent it from being accurate. In contrast, in many situations like ridge regression, incorporating bias permits to drastically reduce variance.It quite depends on the structural assumptions made. For this reason, the worst case result (Theorem 1) is not that informative in my opinion. As they are mainly concerned by the L_1 version of the Wasserstein distance, rather than focussing on the bias, the authors could consider the formulation in terms of inverse cumulative distribution functions in the 1-d setup and the fact that the empirical cdf is never invertible: even if the theoretical cdf is invertible (which naturally guarantees uniqueness of the optimal transport) the underlying mass transportation problem is not as well-posed as that related to its statistical counterpart (however, smoothing the empirical distribution may remedy this issue). The authors propose to use instead the Cramer distance, which is a very popular distance in Statistics and on which many statistical hypothesis testing procedures rely, and review its appealing properties. The comparisons between KL, Wasserstein and Cramer distances is vain in my opinion and willing to come to a general conclusion about the merits of one against the others is naive. In a nonparametric setup, it is always possible to find distributions such that certain of its properties are hidden by certain distances and highlighted by others. This is precisely why you are forced to specify the type of deviations between distributions in nonparametric hypothesis testing (a shift, a change in scale, etc.), there is no way of assessing universally that two distributions are close: optimality can only be assessed for sequences of contiguous hypotheses. The choice of the distance is part of the learning problem.
assistant
 The contribution of the article is related to performance criteria, and in particular to the Wasserstein/Mallows metric, which has received a good deal of attention these last few years in the machine learning literature. The paper starts with a discussion about desirable properties of a loss function and points out the fact that (plug-in) empirical versions of the gradient of this quantity are biased, which limits its interest, insofar as many learning techniques are based on (stochastic) gradient descent. In its current state, this argument looks artificial. Indeed, zero bias can be a desirable properties for an estimate but being biased does not prevent it from being accurate. In contrast, in many situations like ridge regression, incorporating bias permits to drastically reduce variance.It quite depends on the structural assumptions made. For this reason, the worst case result (Theorem 1) is not that informative in my opinion. As they are mainly concerned by the L_1 version of the Wasserstein distance, rather than focussing on the bias, the authors could consider the formulation in terms of inverse cumulative distribution functions in the 1-d setup and the fact that the empirical cdf is never invertible: even if the theoretical cdf is invertible (which naturally guarantees uniqueness of the optimal transport) the underlying mass transportation problem is not as well-posed as that related to its statistical counterpart (however, smoothing the empirical distribution may remedy this issue). The authors propose to use instead the Cramer distance, which is a very popular distance in Statistics and on which many statistical hypothesis testing procedures rely, and review its appealing properties. The comparisons between KL, Wasserstein and Cramer distances is vain in my opinion and willing to come to a general conclusion about the merits of one against the others is naive. In a nonparametric setup, it is always possible to find distributions such that certain of its properties are hidden by certain distances and highlighted by others. This is precisely why you are forced to specify the type of deviations between distributions in nonparametric hypothesis testing (a shift, a change in scale, etc. ), there is no way of assessing universally that two distributions are close: optimality can only be assessed for sequences of contiguous hypotheses. The choice of the distance is part of the learning problem.