Generative Adversarial Networks (GANs) is a powerful family of models that learn an underlying distribution to generate synthetic data. Many existing studies of GANs focus on improving the realness of the generated image data for visual applications, and few of them concern about improving the quality of the generated data for training other classifiers-a task known as the model compatibility problem. Literature also show that some GANs often prefer generating 'easier' synthetic data that are far from the boundaries of the classifiers, and refrain from generating near-boundary data, which are known to play an important roles in training the classifiers. To improve GAN in terms of model compatibility, we propose Boundary-Calibration GANs (BCGANs), which leverage the boundary information from a set of pre-trained classifiers using the original data. In particular, we introduce an auxiliary Boundary-Calibration loss (BC-loss) into the generator of GAN to match the statistics between the posterior distributions of original data and generated data with respect to the boundaries of the pre-trained classifiers. The BC-loss is provably unbiased and can be easily coupled with different GAN variants to improve their model compatibility. Experimental results demonstrate that BCGANs not only generate realistic images like original GANs but also achieves superior model compatibility than the original GANs.
In this work authors consider a problem of 'model compatibility' of GANs, i.e. usefullness of the generated samples for classification tasks. Proposed 'Boundary Calibration' GAN attempts to tackle this issue by adding non-adversarial terms to discriminator, obtained as outputs of the classifiers trained on the original data. For evaluation, it is proposed to compare accuracies obtained by classifiers trained on generated and on real data (termed 'relative acurracy'). Experiments show that the proposed methods improve such scores.
Pros:
- considered problem seems to be important GAN application which has not yet received too much attention.
- proposed method seems to improve the accuracy of classifiers trained on generated data.
Cons:
- the paper is poorly written, has multiple typos and often it is unclear what authors mean.
- the proposed evaluation does not exactly measure the potential improvements from training classifiers with generated data. It would make sense to provide information if generated data can improve classifier's scores, if added to the training data (or small part of it).
- it is unclear whether or not the proposed technique does not affect the sample quality, as no quantitative metrics are provided.
Details:
1. Quantitative scores for quality of generated samples are not provided. Authors instead provide few samples and state that it is difficult to detect the difference. Although sample quality is not the main task here, it certainly is important - otherwise we could train generators solely against the classifier 'boundary calibration' loss terms - this, however would likely lead to adversarial examples. Combining two losses often leads to trade-offs, hence showing that we can improve 'model compatibility' without the loss of sample quality is actually crucial. Metrics such as Inception score [1], FID [2] or KID [3] are desired, especially given relatively poor quality of cifar and mnist samples from all models.
2.The main metric used is the ratio between accurracies of classifiers (trained on real and generated data). It is hence difficult to tell if the classifiers that were used were trained reasonably and achieved reasonable scores.
3. In the abstract, authors claim that 'GANs often prefer generating easier synthetic data that are far from boundaries of the classifiers'. Although for some GAN settings the generators might be biased to do so, in general this claim is unfounded, as GANs optimize divergences that are agnostic to classifier boundaries.
4. It is unclear what kind of classifier-output is used as an input to MMD. Are these continuous logits, discrete class numbers, or one-hot-encoded class identities?
5. Authors use WGAN and MMDGAN with gradient penalty. It is unclear how gradient penalty is applied to MMDGAN as what should be penalized is the witness function, which is different than in WGAN-GP [4], see e.g. [3].
6. It is unclear how embeddings of class information are concatenated to discriminator inputs (p.5).
7. It is unclear to what extent feature selection is deterministic. Authors argue in Section 5.4 that the intersection of top-k features selected from two models should be large. It would be good to provide the same statistics for features selected twice on the same sample.
Overall, the paper currently does not match the quality requirements of ICLR, however it has potential for improvement if the mentioned issues are addressed.
Typos/unclear expressions:
[p1] 'may not willing' >> 'may not be willing'
[p1]'with the property similar to the original data is demanding'  >> properties, demanded/in demand
[p2] 'Although GANs are versatile as aforementioned' - strange wording
[p2] 'The pioneered work' >> 'pioneering work'
[p2] 'information of models' >> 'information from the models'
[p2] effects >> affects
[p3] related works >> related work
[p3] distribution of label >> distribution of labels
[p3] 'generated dataset adopt ' >> 'generated dataset will adopt '
[p3] 'To known about the boundary' >> ' To know the boundary'/'To include the information about the boundary'
[p3] 'the a distance'
[p3] 'the problem to distinguish whether two sets of samples' 
[p4] 'If they are close the sets might be sampled from the same distribution' (?)
[p4] tries to minimized the MMD
[p4] would not leads to
[p6, Table 2 caption] number of estimator used
[p6] at Appendix
[p6] depresses the model compatibility (?)
[p6] can providing
[p7] to known how
[p8] our work open >> our work will open
------------------------------------- Revision -------------------------------------
Although some issue seem to have been clarified, two of my main concerns, i.e. the proposed evaluation and the text quality, have not been resolved. Additionally, as pointed out by reviewer #4, the results seem somewhat incremental. For these reasons, I decided to keep the rating unchanged.