The divergences minimized by non-saturating GAN training
Interpreting generative adversarial network (GAN) training as approximate divergence minimization has been theoretically insightful, has spurred discussion, and has lead to theoretically and practically interesting extensions such as f-GANs and Wasserstein GANs. For both classic GANs and f-GANs, there is an original variant of training and a *non-saturating* variant which uses an alternative form of generator gradient. The original variant is theoretically easier to study, but for GANs the alternative variant performs better in practice. The non-saturating scheme is often regarded as a simple modification to deal with optimization issues, but we show that in fact the non-saturating scheme for GANs is effectively optimizing a reverse KL-like f-divergence. We also develop a number of theoretical tools to help compare and classify f-divergences. We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.
In this paper, firstly, a useful expression of the class of f-divergences is proposed. The authors investigate theoretical properties of some popular f-divergences from newly developed tools. Then, the expression is used to investigate GANs with the non-saturating training scheme. The expressions in Section 3 can be a useful tool to investigate the statistical properties of estimators with f-divergences. However, I think that the usefulness of alternative expressions in Section 3 is not very clear, though an intuitive interpretation is presented. Moreover, the numerical experiment is not sufficient to support some new expressions. Detailed theoretical analysis of the non-saturating training based on the proposed expression would be required for publication from ICLR. In addition, the authors could shorten the paper within 8 pages that is the standard length of ICLR paper. In the paper *On topological properties of f-divergences* (1967), Csiszar intensively studied non-saturating properties of f-divergences in the paper. It would be helpful for readers to add some comments on the relation between the theoretical development in this paper and Csiszar*s paper.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
The divergences minimized by non-saturating GAN training
Interpreting generative adversarial network (GAN) training as approximate divergence minimization has been theoretically insightful, has spurred discussion, and has lead to theoretically and practically interesting extensions such as f-GANs and Wasserstein GANs. For both classic GANs and f-GANs, there is an original variant of training and a *non-saturating* variant which uses an alternative form of generator gradient. The original variant is theoretically easier to study, but for GANs the alternative variant performs better in practice. The non-saturating scheme is often regarded as a simple modification to deal with optimization issues, but we show that in fact the non-saturating scheme for GANs is effectively optimizing a reverse KL-like f-divergence. We also develop a number of theoretical tools to help compare and classify f-divergences. We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



In this paper, firstly, a useful expression of the class of f-divergences is proposed. The authors investigate theoretical properties of some popular f-divergences from newly developed tools. Then, the expression is used to investigate GANs with the non-saturating training scheme. The expressions in Section 3 can be a useful tool to investigate the statistical properties of estimators with f-divergences. However, I think that the usefulness of alternative expressions in Section 3 is not very clear, though an intuitive interpretation is presented. Moreover, the numerical experiment is not sufficient to support some new expressions. Detailed theoretical analysis of the non-saturating training based on the proposed expression would be required for publication from ICLR. In addition, the authors could shorten the paper within 8 pages that is the standard length of ICLR paper. In the paper *On topological properties of f-divergences* (1967), Csiszar intensively studied non-saturating properties of f-divergences in the paper. It would be helpful for readers to add some comments on the relation between the theoretical development in this paper and Csiszar*s paper.
assistant
In this paper, firstly, a useful expression of the class of f-divergences is proposed. The authors investigate theoretical properties of some popular f-divergences from newly developed tools. Then, the expression is used to investigate GANs with the non-saturating training scheme. The expressions in Section 3 can be a useful tool to investigate the statistical properties of estimators with f-divergences. However, I think that the usefulness of alternative expressions in Section 3 is not very clear, though an intuitive interpretation is presented. Moreover, the numerical experiment is not sufficient to support some new expressions. Detailed theoretical analysis of the non-saturating training based on the proposed expression would be required for publication from ICLR. In addition, the authors could shorten the paper within 8 pages that is the standard length of ICLR paper. In the paper *On topological properties of f-divergences* (1967), Csiszar intensively studied non-saturating properties of f-divergences in the paper. It would be helpful for readers to add some comments on the relation between the theoretical development in this paper and Csiszar*s paper.