Generative adversarial networks (GANs) nowadays are capable of producing images of incredible realism. One concern raised is whether the state-of-the-art GAN's learned distribution still suffers from mode collapse. Existing evaluation metrics for image synthesis focus on low-level perceptual quality. Diversity tests of samples from GANs are usually conducted qualitatively on a small scale. In this work, we devise a set of statistical tools, that are broadly applicable to quantitatively measuring the mode collapse of GANs. Strikingly, we consistently observe strong mode collapse on several state-of-the-art GANs using our toolset. We analyze possible causes, and for the first time present two simple yet effective "black-box" methods to calibrate the GAN learned distribution, without accessing either model parameters or the original training data.
This work addresses the important problem of generation bias and a lack of diversity in generative models, which is often called model collapse. It proposed a new metric to measure the diversity of the generative model's "worst" outputs based on the sample clustering patterns. Furthermore, it proposed two blackbox approaches to increasing the model diversity through resampling the latent z. Unlike most existing works that address the model collapse problem, a blackbox approach does not make assumptions about having access to model weights or the artifacts produced during model training, making it more widely applicable than the white-box approaches.  
In terms of experiment setup, the authors chooses face generation as the area to investigate and measures the diversity by detecting the generated face identity. With the proposed methods, the authors showed that most STOA methods have a wide gap between the top p faces of the most popular face identities and randomly sampled faces. It further showed that the proposed blackbox approaches increases the proposed diversity metric without sacrificing image quality.
The proposed diversity measuring metric is lacking both in terms of experimental proofs and intuitive motivations. While the black-box calibration of a GAN model may be attractive under specific settings, the authors did not consider the restrictions under those situations and their design may be hard to implement as a result. For those reasons, I propose to REJECT this paper.
Missing key experiments that will provide more motivation that 1. the new metric reflects human perception of diversity 2. the new metric works better than existing ones:
1. Please provide experiments and/or citation for using the face identity as a proxy for face image diversity. this is important since all your experiments rely on that assumption.
2. Were there experiments that applies your metric to the training datasets like CelebA and FFHQ? In theory your metric should show no gap between N_R_obs and N_R_ref measured on the training dataset since that's the sampled ground truth.
Missing assumptions about blackbox calibration approaches:
1. If we do not have access to the model parameter, the training data, or the artifacts during training like the discriminator, what are some of the real world situations that fit this description? In those cases, is it too much to assume that we can control the random seed input to G? 
2. Is it reasonable to assume some constraints on how much data we can get from the blackbox generator? A website that just exposes the image generation API may not allow you to ping their service 100k times to improve the generation diversity. If you are allowed to do that, it may be reasonable to assume that you can contact the API provider to get access to the rest of the model.
Minor improvements that did not have a huge impact on the score
1. I found the argument about FID in section 2.1 unconvincing. Are there proofs or citations for the claim that real images don't follow multivariate gaussian distribution after applying FID? Copying is indeed an issue that FID cannot detect, but it may be tangential to model collapse for real world concerns like privacy. 
2. The statement "IS, FID and MODE score takes both visual fidelity and diversity into account." under "Evaluation of Mode Collapse" is contradictory to the description in sec 2.1 that IS in fact does not measure diversity. 
3. You may want to consider stating the work as "a pilot study" (sec 6.) earlier in the abstract or in the introduction, so that the reader knows what to expect.