Generative neural networks map a standard, probability distribution to a complex high-dimensional distribution, which represents the real world data set. However, a determinate input distribution as well as a specific architecture of neural networks may impose limitations on capturing the diversity in the high dimensional target space. To resolve this difficulty, we propose a training framework that greedily produce a series of generative adversarial networks that incrementally capture the diversity of the target space. We show theoretically and empirically that our training algorithm converges to the theoretically optimal distribution, the projection of the real distribution onto the convex hull of the network's distribution space.
The paper introduces an incremental training method for GAN's for capturing the diversity of the input space. The paper demonstrate that the proposed method allows smaller distances between the true and generated distribution. I find the idea interesting, but fear that the 60-100 small ensemble models could be replaced by a larger model.
I am curious about why we need incremental training when it seems like we could directly train all the networks jointly. The corresponding generative model is simply stronger so all the convergence arguments would still hold. Is the statistical distance a reasonable estimate for you to determine whether you need an additional generator for incremental training?
Also what are the generator architectures for the experiments? How can you put 60-100 generators within the GPU memory? The latent variable dimension seem to be only 1 for each of your generator? That seems to be seriously handicapping the capacity of each individual generator (to just some data points), so the ensemble distribution might be obtained simply by using a larger dimension z?
There are also other measurements that are used by the GAN community, such as inception score, FID score and samples. It seems also reasonable to verify the effectiveness of this method on CIFAR or LSUN datasets, where the method would have a greater improvement because the data distributions are more complex.
Minor points:
- How do you measure the "Wasserstein distance" for high-dimensional distributions? 
- What not set $\omega_i$ to be always 1? The subsampling process introduced in Algorithm 2 seem to enforce this, and you do this for all the experiments.
- Fix citation typos.
- Fix \mathbf for vector quantities, such as x and z.
- Since the generative models have the same architecture, does the non-convex argument becomes moot when you have a mixture of 2 generators?