Generative adversarial networks (GANs) have enjoyed great success, however often suffer instability during training which motivates many attempts to resolve this issue. Theoretical explanation for the cause of instability is provided in Wasserstein GAN (WGAN), and wasserstein distance is proposed to stablize the training. Though WGAN is indeed more stable than previous GANs, it takes more iterations and time to train. This is because the ways to ensure Lipschitz condition in WGAN (such as weight-clipping) significantly limit the capacity of the network. In this paper, we argue that it is beneficial to ensure Lipschitz condition as well as maintain sufficient capacity and expressiveness of the network. To facilitate this, we develop both theoretical and practical building blocks, using which one can construct different neural networks using a large range of metrics, as well as ensure Lipschitz condition and sufficient capacity of the networks. Using the proposed building blocks, and a special choice of a metric called Dudley metric, we propose Dudley GAN that outperforms the state of the arts in both convergence and sample quality. We discover a natural link between Dudley GAN (and its extension) and empirical risk minimization, which gives rise to generalization analysis.
The authors propose a different type of GAN--the Dudley GAN--that is related to the Dudley metric. In fact, it is very much like the WGAN, but rather than just imposing the function class to have a bounded gradient, they also impose it to be bounded itself. This is argued to be more stable than the WGAN, as gradient clipping is said not necessary for the Dudley GAN. The authors empirically show that the Dudley GAN achieves a greater LL than WGAN for the MNIST and CIFAR-10 datasets.
The main idea [and its variants] looks solid, but with the plethora of GANs in the literature now, after reading I'm still left wondering why this GAN is significantly better than others [BEGAN, WGAN, etc.]. It is clear that imposing the quadratic penalty in equation (3) is really the same constraint as the Dudley norm? The big contribution of the paper seems to be that adding some L_inf regularization to the function class helps preclude gradient clipping, but after reading I'm unsure why this is "the right thing" to do in this case. We know that convergence in the Wasserstein metric is stronger than the Dudley metric, so why is using the weaker metric overweighed by the benefits in training?
Nits: Since the function class is parameterized by a NN, the IPM is not actually the Dudley metric between the two distributions. One would have to show that the NN is dense in Dudley unit ball w.r.t. L_inf norm, but this sort of misnaming had started with the "Wasserstein" GAN.