Generative adversarial networks (GANs) are one of the most popular approaches when it comes to training generative models, among which variants of Wasserstein GANs are considered superior to the standard GAN formulation in terms of learning stability and sample quality. However, Wasserstein GANs require the critic to be 1-Lipschitz, which is often enforced implicitly by penalizing the norm of its gradient, or by globally restricting its Lipschitz constant via weight normalization techniques. Training with a regularization term penalizing the violation of the Lipschitz constraint explicitly, instead of through the norm of the gradient, was found to be practically infeasible in most situations. Inspired by Virtual Adversarial Training, we propose a method called Adversarial Lipschitz Regularization, and show that using an explicit Lipschitz penalty is indeed viable and leads to competitive performance when applied to Wasserstein GANs, highlighting an important connection between Lipschitz regularization and adversarial training.
Final Edit:
I have reviewer the final version of the paper and have decided to increase my score to a weak accept. I maintain some concerns around the empirical evaluation in the paper (collating results from multiple sources with different experimental procedures). But my major concerns have been addressed by the authors in their response and changes to the paper.
----------------------------------
Post rebuttal edit:
Following updates to the paper manuscript, which address my concerns around the correctness of the empirical results, I have updated my review score from 1 to 3.
----------------------------------
Summary:
This paper draws on connections between virtual adversarial training (VAT) and Lipschitz regularization to utilize VAT techniques in the training of WGAN architectures. While this work may be touching on something quite interesting, I felt that the theoretical exposition of the ideas was lacking. The empirical evidence seemed promising in some direction though lacking in others.
I was requested as an emergency reviewer for this paper.
Overview:
This paper is 9 pages length in total. Unfortunately, I felt that the use of an additional page was unwarranted and that the paper contained unnecessary content.
Due to a concern over correctness of some empirical results and issues with the presented derivations I have opted to reject this paper. I hope that these issues can be addressed by the authors in which case I will reassess my score.
1) Under equation 5, "with substantially more stable training behaviour and improved sample quality". A citation should be included for this claim. In fact, recent advances in GAN methods have not required the Wasserstein distance objective [1].
2) I found some issues with Section 2.2.
a) First a comment on related work. There is older work studying the generalization properties of Lipschitz neural networks which is not mentioned in this section. For example, [2]. You also write that learning under Lipschitz constraints became prevalent with the introduction of WGAN. While this is probably true, I think it is fair to point out that many older papers also utilized similar bounds in the vein of improving generalization. For example [3], which also aimed to limit the gradient norm of deep neural networks.
b) I felt that this subsection was a little bloated and the content did not fit entirely under the heading. A large chunk of this section is dedicated to discussing potential issues arising with the gradient penalty formulation of WGAN and alternative approaches such as the Banach WGAN. While these are useful additions they did not feel critical to this work and in my opinion did not deserve an extension beyond the 8 page recommendation.
3) I found the discussion in Section~3 a little difficult to follow. I will summarize my key concerns below.
a) The authors assume that generalizing Lipschitz continuity to a premetric space is trivial. While the more important results seems believable I am not convinced by the presentation of these results and would prefer to have seen this given more careful treatment. For example, premetrics need not obey symmetry or the triangle inequality (assuming this is the definition used by the authors --- including this would be valuable). It is written (paraphrasing) that a mapping $f$ is $K$-Lipschitz iff for any $x$ the supremum over $r$ is bounded by $K$. However, $r$ only appears on the right-hand side of a potentially asymmetric distance function. Moreover, many properties of Lipschitz continuous functions depend on the triangle inequality holding in the metric space and these would fail to hold here.
b) When connecting ALP to VAT some of the differences in the formulations are hand-waved away in unconvincing ways. Under the trivial metric, the Lipschitz constant is given by the maximum distance in the output space. With this observation, it seems trivial that the VAT formulation will perform a form of Lipschitz regularization. However, the Lipschitz constant does not take into account distance in the input space in a meaningful way and so I am unsure to what extend the connection is really meaningful. Further, the $r < \epsilon$ constraint in the VAT model is treated as an inconvenient implementation detail but I am not convinced this is sufficient. Indeed, this $\epsilon$ could be used to bound the input deviations and thus could be seen as affecting the Lipschitz constant under a more reasonable metric.
4) I felt that Section~4.1 presented an important discussion coupled with an interesting toy problem to highlight benefits and shortcomings of the proposed method. However, this section was moderately long and contributed only a little towards understanding the practical settings users of ALR would care about in practice. I did not gain much intuition into how ALR might generalize to high dimensional settings and was concerned by the fact that the distribution $P_\epsilon$ used was heavily hand-engineered and did not match up with the ones used in later experiments.
a) I did not understand the comments that constraining the Lipschitz constant globally may be undesirable in WGANs. The dual optimization problem requires a Lipschitz constraint be enforced over the support of the distributions and we should not care outside of this region in any case (except in cases where the generator might move the support to a currently under-regularized region of the critic domain in which case a global constraint may be advantageous).
5) I am not particularly up to date with evaluation of GAN models but to me the presented results in the main paper looked mostly reasonable. Some major concerns did remain to me which I would appreciate being addressed by the authors.
a) I have one question on the reported "Best" inception score for the WGAN-ALP and Progressive-GAN models. You stated that each model was trained 5 times and reported the mean, standard deviation and best results. However, the difference between the best and average scores alone would constitute a higher standard deviation: $\sqrt{(8.80 - 8.56)^2 / 4} = 0.12$. Please can you clarify how exactly each of these values was computed?
b) In the main paper the authors write that ALP is able to work in high dimensional settings (though is not competitive with state of the art). In the appendix however the authors point out that they must make significant modifications to the training objective by including a squared Lipschitz constraint violation term (violating further the comparison to VAT). I do not consider this a huge issue, but it should be discussed in the main paper.
c) Finally, the authors employed a range of different hyperparameter settings through their experiments but gave little guidance on how to choose these settings in practice or how sensitive their proposed method is to changes in these hyperparameters. I believe that this would be a highly-valuable addition to the paper and would help distinguish this method from other training stabilization proposals.
Minor comments:
- In paragraph 1, you write that WGAN requires critic to consist of only 1-Lipschitz functions. This is true of the Wasserstein distance estimation problem but the WGAN only requires the correct gradient direction (scaling of the critic is fine).
- In summary points in intro, "ALR" is used before acronym is defined.
- Equation (12) and (13) are twice normalized (||r_k||^2=1 by definition). Similar issue in (22) and (23).
- First para of Section 3, you write "on the space of labels". Do you mean on the probability simplex?
References:
[1] "Large scale GAN training for high fidelity natural image synthesis", Brock, Donahue, and Simonyan
[2] "The sample complexity of pattern classification with neural networks", Bartlett
[3] "Double backpropagation increasing generalization performance", Drucker and LeCun