Demystifying MMD GANs
We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramér GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.
 This paper claims to demystify MMD-GAN, a generative adversarial network with the maximum mean discrepancy (MMD) as a critic, by showing that the usual estimator for MMD yields unbiased gradient estimates (Theorem 1). It was noted by the authors that biased gradient estimate can cause problem when performing stochastic gradient descent, as also noted previously by Bellemare et al. The authors also proposed a kernel inception distance (KID) as a quantitative evaluation metric for GAN. The KID is defined to be the squared MMD between inception representation of the distributions. In experiments, the authors compared the quality of samples generated by MMD-GAN with various kernels with the ones generated from WGAN-GP (Gulrajani et al., 2017) and Cramer GAN (Bellemare et al., 2017). The empirical results show the benefits of using the MMD on top of deep convolutional features. The major flaw of this paper is that its contribution is not really clear. Showing that the expectation and gradient can be interchanged (Theorem 1) does not seem to provide sufficient significance. Unbiasedness of the gradient alone does not guarantee that training will be successful and that the resulting models will better reflect the underlying data distribution, as evident by other successful variants of GANs, e.g., WGAN, which employ biased estimate. Indeed, since the training process relies on a small mini-batch, a small bias could help counteract the potentially high variance of the gradient estimate. The key is rather a good balance of both bias and variance during the training process and a guarantee that the estimate is asymptotically unbiased wrt the training iterations. Lastly, I do not see how the empirical results would demystify MMD-GANs, as claimed by the paper. The paper is clearly written. Some minor comments: - The proof of the main result, Theorem 1, should be placed in the main paper. - Page 7, 2nd paragraph: later --> layer
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Demystifying MMD GANs
We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramér GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper claims to demystify MMD-GAN, a generative adversarial network with the maximum mean discrepancy (MMD) as a critic, by showing that the usual estimator for MMD yields unbiased gradient estimates (Theorem 1). It was noted by the authors that biased gradient estimate can cause problem when performing stochastic gradient descent, as also noted previously by Bellemare et al. The authors also proposed a kernel inception distance (KID) as a quantitative evaluation metric for GAN. The KID is defined to be the squared MMD between inception representation of the distributions. In experiments, the authors compared the quality of samples generated by MMD-GAN with various kernels with the ones generated from WGAN-GP (Gulrajani et al., 2017) and Cramer GAN (Bellemare et al., 2017). The empirical results show the benefits of using the MMD on top of deep convolutional features. The major flaw of this paper is that its contribution is not really clear. Showing that the expectation and gradient can be interchanged (Theorem 1) does not seem to provide sufficient significance. Unbiasedness of the gradient alone does not guarantee that training will be successful and that the resulting models will better reflect the underlying data distribution, as evident by other successful variants of GANs, e.g., WGAN, which employ biased estimate. Indeed, since the training process relies on a small mini-batch, a small bias could help counteract the potentially high variance of the gradient estimate. The key is rather a good balance of both bias and variance during the training process and a guarantee that the estimate is asymptotically unbiased wrt the training iterations. Lastly, I do not see how the empirical results would demystify MMD-GANs, as claimed by the paper. The paper is clearly written. Some minor comments: - The proof of the main result, Theorem 1, should be placed in the main paper. - Page 7, 2nd paragraph: later --> layer
assistant
 This paper claims to demystify MMD-GAN, a generative adversarial network with the maximum mean discrepancy (MMD) as a critic, by showing that the usual estimator for MMD yields unbiased gradient estimates (Theorem 1). It was noted by the authors that biased gradient estimate can cause problem when performing stochastic gradient descent, as also noted previously by Bellemare et al. The authors also proposed a kernel inception distance (KID) as a quantitative evaluation metric for GAN. The KID is defined to be the squared MMD between inception representation of the distributions. In experiments, the authors compared the quality of samples generated by MMD-GAN with various kernels with the ones generated from WGAN-GP (Gulrajani et al., 2017) and Cramer GAN (Bellemare et al., 2017). The empirical results show the benefits of using the MMD on top of deep convolutional features. The major flaw of this paper is that its contribution is not really clear. Showing that the expectation and gradient can be interchanged (Theorem 1) does not seem to provide sufficient significance. Unbiasedness of the gradient alone does not guarantee that training will be successful and that the resulting models will better reflect the underlying data distribution, as evident by other successful variants of GANs, e.g., WGAN, which employ biased estimate. Indeed, since the training process relies on a small mini-batch, a small bias could help counteract the potentially high variance of the gradient estimate. The key is rather a good balance of both bias and variance during the training process and a guarantee that the estimate is asymptotically unbiased wrt the training iterations. Lastly, I do not see how the empirical results would demystify MMD-GANs, as claimed by the paper. The paper is clearly written. Some minor comments: - The proof of the main result, Theorem 1, should be placed in the main paper. - Page 7, 2nd paragraph: later --> layer