Generative adversarial networks (GANs) are capable of producing high quality samples, but they suffer from numerous issues such as instability and mode collapse during training. To combat this, we propose to model the generator and discriminator as agents acting under local information, uncertainty, and awareness of their opponent. By doing so we achieve stable convergence, even when the underlying game has no Nash equilibria. We call this mechanism implicit competitive regularization (ICR) and show that it is present in the recently proposed competitive gradient descent (CGD). When comparing CGD to Adam using a variety of loss functions and regularizers on CIFAR10, CGD shows a much more consistent performance, which we attribute to ICR. In our experiments, we achieve the highest inception score when using the WGAN loss (without gradient penalty or weight clipping) together with CGD. This can be interpreted as minimizing a form of integral probability metric based on ICR.
https://openreview.net/pdf?id=SkxaueHFPB
The paper has some interesting ideas but I don’t think any of them are fully fleshed out.
I find the reporting of Inception Score highly suspect. The authors choose WGAN-GP as a baseline and report scores of ~4.5 vs ~5.5 with their modification. However the WGAN-GP paper reports an IS of 7.86 on CIFAR. Furthermore, current GAN SOTA on CIFAR is approaching IS=9. I am not making the argument that the authors ought to demonstrate SOTA results, however they should at least present results which are consistent with the published results of their chosen baseline.
The authors then make this statement:
“Thus, its superior performance supports the claim that ICR is the appropriate form of regularization for GANs. We emphasize that in our experiments we did not perform any architecture or hyperparameter tuning, and instead use a model intended to be used with WGAN gradient penalty”
This does not hold, since the numbers reported are far below the actual baseline.
Besides this major point, I am unconvinced by some of the mathematical statements in the paper. Much of the mathematical details are deferred to the original CGD paper. It is not really particularly reader-friendly to defer that to the CGD paper since they are seemingly crucial to the discussion here. Relative to the CGD paper some signs have been flipped and some definitions appear to be used in subtly different ways which makes for a very difficult read. I feel that far too much has been left as an exercise to the reader.
Concretely my concerns refer to the main discussion of the effect of the CGD as a regularizer:
The authors state:
“If some of the singular values of Dxy are very large, this amounts to approximately restricting the update to the orthogonal complement of the corresponding singular vectors” 
I don’t see how this is the case. The terms Dxy/Dyx aren’t really introduced or defined anywhere in this work. Assuming is the transpose of the other (?) then the update direction is:
A + B where A=inv(S) grad_x and B = inv(S)Dxy grad_y (and S = I + Dxy Dyx). So we have a term which is being affected by the smallest singular values of S and a term which is the orthogonal projection of grad_y onto Dxy, alternatively the ridge-regression fit of grad_y on Dxy which would attenuate directions corresponding to the *small* singular values (as is well known from the theory of ridge regularizers). I feel like there is much more to say here than what is discussed in the paper in very vague terms. 
Of course the effective rank of S, or the rate of decay of its singular values is crucially important. In practise I would assume the smaller SVs of Dxy to be difficult to estimate or the matrix to be rank deficient in which case they would simply be unity in the inverse whereas the directions corresponding to large singular values would be attenuated. So in this case it is the regularized orthogonal complement but its not clear (if the matrix is not full rank) that it is a meaningful direction (and again this is all highly dependent on the effective rank, too).
Further on it is mentioned: “For smoothly varying singular vectors, this can be though of as approximately constraining the trajectories to a manifold of robust play”.
First it is not at all clear to me what “smoothly varying singular vectors” are. Varying with respect to what? Secondly, the “manifold of robust play” has not been defined anywhere. 
Finally, figure 3 is quite bizarre to me. None of the quantities have been rigorously defined and so it seems like the relative effect of each of the arrows and the manifold have been drawn arbitrarily in order to fit the story, rather than to actually illuminate the true behaviour in an intuitive manner. 
B (defined above) has a very clear interpretation as a least-squares fit so I figure that any geometric interpretation of the CGD update direction could start from there.