The Numerics of GANs
In this paper, we analyze the numerics of common algorithms for training Generative Adversarial Networks (GANs). Using the formalism of smooth two-player games we analyze the associated gradient vector field of GAN training objectives. Our findings suggest that the convergence of current algorithms suffers due to two factors: i) presence of eigenvalues of the Jacobian of the gradient vector field with zero real-part, and ii) eigenvalues with big imaginary part. Using these findings, we design a new algorithm that overcomes some of these limitations and has better convergence properties. Experimentally, we demonstrate its superiority on training common GAN architectures and show convergence on GAN architectures that are known to be notoriously hard to train.
This paper presents a novel analysis of the typical optimization algorithm used in GANs (simultaneous gradient ascent) and identifies problematic failures when the Jacobian has large imaginary components or zero real components. Motivated by these failures, they present a novel consensus optimization algorithm for training GANs. The consensus optimization is validated on a toy MoG dataset as well as CIFAR-10 and CelebA in terms of sample quality and inception score.
I found this paper enjoyable to read and the results compelling. My primary concern is the lack of hyperparameter search when comparing optimization algorithms and lack of evidence that the problems identified with simultaneous gradient ascent are truly problems in practice.
Major concerns:
1. The constant discriminator/generator loss is very weird. Any ideas on what is going on? Does this hold when you swap in different divergences, e.g. WGAN?
2. Which losses were you using for the generator/discriminator? From Fig 4, it looks like you’re using the modified generator objective that deviates from the minimax game. Do your convergence results hold for non-zero sum games when f and g are different functions, i.e. f != -g?
3. Experiments: You note that simultaneous and alternating gradient descent may require tiny step sizes for convergence. In your experiments did you try tuning learning rates? Additional optimization hyperparameters? Without sweeping over learning rates it seems unfair to say that one optimization technique is better than another. If consensus optimization is more robust across learning rates then that should be demonsrated empirically.
4. Empirical validation of the failures of simultaneous gradient ascent: are zero real components and large imaginary parts really the problem in practice? You never evaluate this. I expected to see empirical validation in the form of plots showing the distribution of eigenvalues of the Jacobian over the course of optimization for simultaneous gradient ascent.
Minor concerns:
1. From the perspective of consensus optimization and \gamma being a Lagrange multiplier, I’d expect to see \gamma *adapt* over time. In particular, ADMM-style algorithms for consensus optimization have dynamics on \gamma as well. Have you explored these variants?
2. Many recent approaches to optimization in GANs have proposed penalties on gradients (typically the gradient of the discriminator w.r.t. Its inputs as in WGAN-GP). Are these related to the approach you propose, and if so is another explanation for the success of consensus optimization just that you’re encouraging a smooth discriminator? One way of testing this would be to remove the additional \nabla L(x) term for the generator update, but keep it for the discriminator update.
3. Figure 5 (table comparing architectures and optimizers) is a nice demonstration of the stability of Consensus relative to other approaches, but did you sweep over learning rate/optimization hyperparameters? If so, then I think this would make a strong figure to include in the main text.
4. I had a hard time following the convergence theory section and understanding how it relates to the results we see in practice.