GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Generative Adversarial Networks (GANs) excel at creating realistic images with complex models for which maximum likelihood is infeasible. However, the convergence of GAN training has still not been proved. We propose a two time-scale update rule (TTUR) for training GANs with stochastic gradient descent on arbitrary GAN loss functions. TTUR has an individual learning rate for both the discriminator and the generator. Using the theory of stochastic approximation, we prove that the TTUR converges under mild assumptions to a stationary local Nash equilibrium. The convergence carries over to the popular Adam optimization, for which we prove that it follows the dynamics of a heavy ball with friction and thus prefers flat minima in the objective landscape. For the evaluation of the performance of GANs at image generation, we introduce the 'Fréchet Inception Distance" (FID) which captures the similarity of generated images to real ones better than the Inception Score. In experiments, TTUR improves learning for DCGANs and Improved Wasserstein GANs (WGAN-GP) outperforming conventional GAN training on CelebA, CIFAR-10, SVHN, LSUN Bedrooms, and the One Billion Word Benchmark.
Generative adversarial networks (GANs) are turning out to be a very important advance in machine learning. Algorithms for training GANs have difficulties with convergence. The paper proposes a two time-scale update rule (TTUR) which is shown (proven) to converge under certain assumptions. Specifically, it shows that GAN Adam updates with TTUR can be expressed as ordinary differential equations, and therefore can be proved to converge using a similar approach as in Borkar' 1997 work. The recommendation is to use two different update rules for generator and discriminator, with the latter being faster, in order to have convergence guarantees.
***Concerns:
      
1-Assumptions A1 to A6 and the following Theorem 1 are symmetric wrt h and g, thus one can assume it makes no difference which one of the networks (discriminator or generator) has the faster updates. A discussion on why this symmetry does not hold in practice, as pointed out by the authors in introduction, is necessary.
      
2-The assumptions used in the proof should be investigated in each of the provided experiments to make sure the proof is in fact useful, for example one of the key assumptions that a = o(b) is clearly not true in experiments (they use a = O(a) instead).
      
3-The less variation pointed out in Figure 2 are probably due to lower learning rate of the generator in TTUR compared to SGD, however since the learning rate values are not reported for Figure 2, we can not be sure.
      
4-In Figure 6, variance and/or confidence intervals should be reported.
      
5-The reported learning rate for TTUR appears to be only linearly faster for the discriminator (3 times faster in Figure 6, 1.25 times faster in Figure 3). This is not consistent with the assumption A2. Moreover, it is well known that a much faster discriminator for regular GANs undermines the generator training. However, according to Theorem 1, the system should converge. This point needs to be addressed.
      
6-There is insufficient previous/related work discussion, specifically about the extensive use of two time updates in dynamic systems, as well as the research on GAN convergence.
      
7-In general, theorems, specifically Theorem 2, are not well explained and elaborated on. The exposited ideas can become more clear and the paper more readable with a line or two of explanation in each case.
      
In summary, while the proofs for convergence are noteworthy, a more detailed investigation and explanation of the effect of assumptions in each of the experiments, and how deviating from them can hurt the convergence in practice, will help clarify the limitations of the proof.