Generative adversarial network (GAN) is one of the best known unsupervised learning techniques these days due to its superior ability to learn data distributions. In spite of its great success in applications, GAN is known to be notoriously hard to train. The tremendous amount of time it takes to run the training algorithm and its sensitivity to hyper-parameter tuning have been haunting researchers in this area. To resolve these issues, we need to first understand how GANs work. Herein, we take a step toward this direction by examining the dynamics of GANs. We relate a large class of GANs including the Wasserstein GANs to max-min optimization problems with the coupling term being linear over the discriminator. By developing new primal-dual optimization tools, we show that, with a proper stepsize choice, the widely used first-order iterative algorithm in training GANs would in fact converge to a stationary solution with a sublinear rate. The same framework also applies to multi-task learning and distributional robust learning problems. We verify our analysis on numerical examples with both synthetic and real data sets. We hope our analysis shed light on future studies on the theoretical properties of relevant machine learning problems.
This paper studies the convergence of a primal-dual algorithm on a certain min-max problem and experimentally shows that it works in GANs and multi-task learning.
This paper is clear and well-written. The convergence guarantee looks neat, and convergence to stationary points is a sensible thing on non convex-concave problems. I am not super familiar with the literature of saddle-point optimization and may not have a good sense about the significance of the theoretical result.
My main concern is that the assumptions in the theory are rather over-restrictive and it’s not clear what intuitions or new messages they bring in for the practice of GANs. The convergence theorem requires the maximization problem (over discriminators) to be strictly concave. On GANs, this assumption is not (near) satisfied beyond the simple case of the LQG setting (PSD quadratic discriminators). On the other hand, the experiments on GANs just seem to say that the algorithm works but not much more beyond that. There is a brief discussion about the improvement in time consumption but it doesn’t have a report a quantitative comparison in the wall time.
On multi-task learning, the proposed algorithm shows improvement over the baseline. However it is also unclear whether it is the *formulation* (12) that brings in the improvement, or it is the actual primal-dual *algorithm*. Perhaps it might be good to try gradient descent on (12) and see if it also works well. 
In general, I would recommend the authors to have a more convincing demonstration of the strength of this algorithm over baseline methods on min-max problems, either theoretical or empirical.