IMPROVING MMD-GAN TRAINING WITH REPULSIVE LOSS FUNCTION
Generative adversarial nets (GANs) are widely used to learn the data sampling process and their performance may heavily depend on the loss functions, given a limited computational budget. This study revisits MMD-GAN that uses the maximum mean discrepancy (MMD) as the loss function for GAN and makes two contributions. First, we argue that the existing MMD loss function may discourage the learning of fine details in data as it attempts to contract the discriminator outputs of real data. To address this issue, we propose a repulsive loss function to actively learn the difference among the real data by simply rearranging the terms in MMD. Second, inspired by the hinge loss, we propose a bounded Gaussian kernel to stabilize the training of MMD-GAN with the repulsive loss function. The proposed methods are applied to the unsupervised image generation tasks on CIFAR-10, STL-10, CelebA, and LSUN bedroom datasets. Results show that the repulsive loss function significantly improves over the MMD loss at no additional computational cost and outperforms other representative loss functions. The proposed methods achieve an FID score of 16.21 on the CIFAR-10 dataset using a single DCGAN network and spectral normalization.
OVERALL COMMENTS:
I haven't had much time to write this, so I'm giving a low confidence score and you should feel free to correct me.
I didn't think this paper was very clear. 
I had trouble grasping what the contributions were supposed to be
and I had trouble judging the significance of the experiments. 
That said, now that (I think) I understand what's going on,
the idea seems well motivated, the connection between the repulsion and the use of label information in other
GAN variants makes sense to me, and the statements you are making seem (as much as I had time to check them) correct. 
This leaves the issue of scientific significance. 
I feel like I need to understand what specifically contributed to the improvements in table 1 to evaluate significance. 
First of all, it seems like there are a lot of other 'good-scoring' models left out of this table. 
I understand that you make the claim that your improvement is orthogonal, but that seems like something that needs to
be tested empirically. You have orthogonal motivation but it might be that in practice your technique works for a reason
similar to the reason other techniques work. I would like to see more exploration of this. 
Second, are the models below the line the only models using spectral norm? I can't tell.
Overall, it's hard for me to envision this work really seriously changing the course of research on GANs,
but that's perhaps too high a bar for poster acceptance.
For these reasons, I am giving a score of 6.
DETAILED COMMENTS ON TEXT:
> their performance heavily depends on the loss functions used in training.
This is not true, IMO. See [1]
> may discourage the learning of data structures
What does 'data structures' mean in this case?
It has another more common usage that makes this confusing.
> Several loss functions have been proposed
IMO this list doesn't belong in the main body of the text.
I would move it to an appendix.
> We assume linear activation is used at the last layer of D
I'm not sure what this means?
My best guess is just that you're saying there is no activation function applied to the logits.
> Arjovsky et al. (2017) showed that, if the supports of PX and PG do not overlap, there exists a perfect discriminator...
This doesn't affect your paper that much, but was this really something that needed to be shown?
If the discriminator has finite capacity it's not true in general and if it has infinite capacity its vacuous.
> We propose a generalized power iteration method...
Why do this when we can explicitly compute the singular values as in [2]?
Genuine question.
> MS-SSIM is not compatible with CIFAR-10 and STL-10 which have data from many classes;
Just compute the intra-class MS-SSIM as in [3].
> Higher IS and lower FID scores indicate better image quality
I'm a bit worried about using the FID to evaluate a model that's been trained w/ an MMD loss where 
the discriminator is itself a neural network w/ roughly the same architecture as the pre-trained image classifier
used to compute the FID. What can you say about this?
Am I wrong to be worried?
> Table 1: 
Which models use spectral norm?
My understanding is that this has a big influence on the scores.
This seems like a very important point.
REFERENCES:
[1] Are GANs Created Equal? A Large-Scale Study
[2] The Singular Values of Convolutional Layers
[3] Conditional Image Synthesis With Auxiliary Classifier GANs