DISCRIMINATOR REJECTION SAMPLING
We propose a rejection sampling scheme using the discriminator of a GAN to approximately correct errors in the GAN generator distribution. We show that under quite strict assumptions, this will allow us to recover the data distribution exactly. We then examine where those strict assumptions break down and design a practical algorithm-called Discriminator Rejection Sampling (DRS)-that can be used on real data-sets. Finally, we demonstrate the efficacy of DRS on a mixture of Gaussians and on the state of the art SAGAN model. On ImageNet, we train an improved baseline that increases the best published Inception Score from 52.52 to 62.36 and reduces the Fréchet Inception Distance from 18.65 to 14.79. We then use DRS to further improve on this baseline, improving the Inception Score to 76.08 and the FID to 13.75.
his paper assumes that, in a GAN, the generator is not perfect and some information is left in the discriminator, so that it can be used to 'reject' some of the 'fake' examples produced by the generator.
The introduction, problem statement and justification for rejection sampling are excellent, with a level of clarity that makes it understandable by non expert readers, and a wittiness that makes the paper fun to read. I assume this work is novel: the reviewer is more an expert in rejection than in GANs, and is aware how few publications rely on rejection.
However, the authors fail to compare their algorithm to a much simpler rejection scheme, and a revised version should discuss this issue.
Let's jump to equation (8): compared to a simple use of the dicriminator for rejection, it adds the term under the log.
The basic rejection equation would read F(x) = D*(x) - gamma and one would adjust the threshold gamma to obtain the desired operating point. I am wondering why no comparison is provided with basic rejection? 
Let me try to understand the Gaussian mixture experiment, as the description is ambiguous:
- GAN setting: 10K examples are generated and reported in figure 3?
- DRS setting: 10K examples are generated, and submitted to algorithm in figure 1. For each batch, a line search sets gamma so that 95% of the examples are accepted. Thus only 9.5K are reported in figure 3.
- What about basic rejection using F(x) = D*(x) - gamma: how does it compare to DRS at the same 95% accept?
If this is my understanding, then the comparison in Figure 3 in unfair, as DRS is allowed to pick and choose.
For completeness, basic rejection should also be added.
Going back to Eq.(8), one realizes that the difference between DRS rejection and basic rejection may be negligible.
First order Taylor expansion of log(1-x) that would apply to the case where the rejection probability is small yields:
F(x) = (D*(x) - D*_M) + exp(D*(x) - D*_M) 
x+ exp(x) is monotonous, so thresholding over it is the same as thresholding over x: back to basic rejection!