AdaGAN: Boosting Generative Models
Generative Adversarial Networks (GAN) are an effective method for training generative models of complex data such as natural images. However, they are notoriously hard to train and can suffer from the problem of missing modes where the model is not able to produce examples in certain regions of the space. We propose an iterative procedure, called AdaGAN, where at every step we add a new component into a mixture model by running a GAN algorithm on a re-weighted sample. This is inspired by boosting algorithms, where many potentially weak individual predictors are greedily aggregated to form a strong composite predictor. We prove analytically that such an incremental procedure leads to convergence to the true distribution in a finite number of steps if each step is optimal, and convergence at an exponential rate otherwise. We also illustrate experimentally that this procedure addresses the problem of missing modes.
AdaGAN is a meta-algorithm proposed for GAN. The key idea of AdaGAN is: at each step reweight the samples and fits a generative model on the reweighted samples. The final model is a weighted addition of the learned generative models. The main motivation is to reduce the mode-missing problem of GAN by reweighting samples at each step.
It is claimed in the Introduction that AdaGAN can also use WGAN or mode-regularized GAN as base generators (line 55). This claim seems to be an overstatement. The key assumption of AdaGAN is that the base generator aims at minimizing the f-divergence (as mentioned in Line 136-137). This assumption does not hold for WGAN or mode-regularized GAN: WGAN minimizes the Wasserstein distance, and the mode-regularized GAN has an additional mode regularizer in the objective.
WGAN and mode-regularized GAN are state-of-the-art generative models targeting the mode-missing problem. It is more convincing if the author can demonstrate that AdaGAN outperforms the two algorithms.
Corollary 1 and 2 assume that the support of learned distribution P_g and the true distribution P_d should overlap for at least 1-beta. This is usually not true in practice. A common situation when training GAN is that the data manifold and the generation manifold are disjoint (see e.g., the WGAN paper).