Understand the dynamics of GANs via Primal-Dual Optimization
Generative adversarial network (GAN) is one of the best known unsupervised learning techniques these days due to its superior ability to learn data distributions. In spite of its great success in applications, GAN is known to be notoriously hard to train. The tremendous amount of time it takes to run the training algorithm and its sensitivity to hyper-parameter tuning have been haunting researchers in this area. To resolve these issues, we need to first understand how GANs work. Herein, we take a step toward this direction by examining the dynamics of GANs. We relate a large class of GANs including the Wasserstein GANs to max-min optimization problems with the coupling term being linear over the discriminator. By developing new primal-dual optimization tools, we show that, with a proper stepsize choice, the widely used first-order iterative algorithm in training GANs would in fact converge to a stationary solution with a sublinear rate. The same framework also applies to multi-task learning and distributional robust learning problems. We verify our analysis on numerical examples with both synthetic and real data sets. We hope our analysis shed light on future studies on the theoretical properties of relevant machine learning problems.
 This paper analyses the learning dynamics of GANs by formulating the problem as a primal-dual optimisation problem. This formulation assumes a limited class of models -- Wasserstein GANs with discriminators using linear combinations of base functions. Although this setting is limited, it advanced our understanding of a central problem related to GANs, and provides intuition for more general cases. The paper further shows the same analysis can be applied to multi-task learning and distributed learning. Pros: * The paper is well written and well motivated * The theoretical analysis is solid and provide intuition for more complex problems Cons: * The primal-dual formulation assumes Wasserstein GANs using linear discriminator. This simplification is understandable, but it would be helpful to at least comment on more general cases. * Experiments are limited: only results from GANs with LQG setting were presented. Since the assumption of linear discriminator (in basis) is already strong, it would be helpful to show the experimental results from this more general setting. * The results on multi-task learning were interesting, but the advantage of optimising the mixing weights was unclear compared with the even mixture baseline. This weakens the analysis of the learning dynamics, since learning the mixing did not seem to be important. It would also be helpful to comment on recently proposed stabilising methods. For example, would spectral normalisation bring learning dynamics closer to the assumed model?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Understand the dynamics of GANs via Primal-Dual Optimization
Generative adversarial network (GAN) is one of the best known unsupervised learning techniques these days due to its superior ability to learn data distributions. In spite of its great success in applications, GAN is known to be notoriously hard to train. The tremendous amount of time it takes to run the training algorithm and its sensitivity to hyper-parameter tuning have been haunting researchers in this area. To resolve these issues, we need to first understand how GANs work. Herein, we take a step toward this direction by examining the dynamics of GANs. We relate a large class of GANs including the Wasserstein GANs to max-min optimization problems with the coupling term being linear over the discriminator. By developing new primal-dual optimization tools, we show that, with a proper stepsize choice, the widely used first-order iterative algorithm in training GANs would in fact converge to a stationary solution with a sublinear rate. The same framework also applies to multi-task learning and distributional robust learning problems. We verify our analysis on numerical examples with both synthetic and real data sets. We hope our analysis shed light on future studies on the theoretical properties of relevant machine learning problems.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper analyses the learning dynamics of GANs by formulating the problem as a primal-dual optimisation problem. This formulation assumes a limited class of models -- Wasserstein GANs with discriminators using linear combinations of base functions. Although this setting is limited, it advanced our understanding of a central problem related to GANs, and provides intuition for more general cases. The paper further shows the same analysis can be applied to multi-task learning and distributed learning. Pros: * The paper is well written and well motivated * The theoretical analysis is solid and provide intuition for more complex problems Cons: * The primal-dual formulation assumes Wasserstein GANs using linear discriminator. This simplification is understandable, but it would be helpful to at least comment on more general cases. * Experiments are limited: only results from GANs with LQG setting were presented. Since the assumption of linear discriminator (in basis) is already strong, it would be helpful to show the experimental results from this more general setting. * The results on multi-task learning were interesting, but the advantage of optimising the mixing weights was unclear compared with the even mixture baseline. This weakens the analysis of the learning dynamics, since learning the mixing did not seem to be important. It would also be helpful to comment on recently proposed stabilising methods. For example, would spectral normalisation bring learning dynamics closer to the assumed model?
assistant
 This paper analyses the learning dynamics of GANs by formulating the problem as a primal-dual optimisation problem. This formulation assumes a limited class of models -- Wasserstein GANs with discriminators using linear combinations of base functions. Although this setting is limited, it advanced our understanding of a central problem related to GANs, and provides intuition for more general cases. The paper further shows the same analysis can be applied to multi-task learning and distributed learning. Pros: * The paper is well written and well motivated * The theoretical analysis is solid and provide intuition for more complex problems Cons: * The primal-dual formulation assumes Wasserstein GANs using linear discriminator. This simplification is understandable, but it would be helpful to at least comment on more general cases. * Experiments are limited: only results from GANs with LQG setting were presented. Since the assumption of linear discriminator (in basis) is already strong, it would be helpful to show the experimental results from this more general setting. * The results on multi-task learning were interesting, but the advantage of optimising the mixing weights was unclear compared with the even mixture baseline. This weakens the analysis of the learning dynamics, since learning the mixing did not seem to be important. It would also be helpful to comment on recently proposed stabilising methods. For example, would spectral normalisation bring learning dynamics closer to the assumed model?