Answers.Body: <p>The <a href="https://en.wikipedia.org/wiki/Lipschitz_continuity" rel="noreferrer">Lipschitz constraint</a> is essentially that a function must have a maximum gradient. The specific maximum gradient is a hyperparameter.</p>
  <p>It's <strong>not</strong> mandatory for a discriminator to obey a Lipschitz constraint. However, in <a href="https://arxiv.org/pdf/1701.07875.pdf" rel="noreferrer">the WGAN paper</a> they find that <em>if</em> the discriminator <em>does</em> obey a Lipschitz constraint, the GAN works much better.</p>
  <p>A perfect discriminator would perfectly accept all the real samples (output=1 with low gradient) and perfectly reject all the fake samples (output=0 with low gradient), but this provides hardly any gradient information to help train the generator. By limiting the gradient of the discriminator, they force it to be a worse discriminator, but provide more gradient information which helps train the generator.</p>
  <p>You can see this in figure 2 (page 9) of the WGAN paper. The red line is a good discriminator but its gradient is nearly 0 at most points. The cyan line (which for some reason is presented upside-down) is clearly much worse as a discriminator, but is much better for training the generator because its gradient is not zero.</p>
  <p>The way they limit the discriminator's <em>overall</em> gradient in the WGAN paper is by separately limiting (clipping) each of the weights in the discriminator. They are aware this is a &quot;terrible&quot; idea, and leave better ideas for further research. The paper you are asking about is one of those better ideas.</p>
  <p>Note that since back-propagation works backwards, enforcing a <em>maximum</em> gradient (Lipschitz continuity) on the discriminator in the <em>forward</em> direction causes it to have a <em>minimum</em> gradient in the <em>backward</em> direction, which is what we want when training the generator.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-22T08:14:12.393Z
Answers.Text: can you explain more about why enforcing a maximum gradient on the discriminator in forwarding leads to the minimum gradient in the backward direction in generator? And why why small gradient is good for generator?
Answers.UserDisplayName: CuCaRot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-23T22:54:56.653Z
Answers.Text: @CuCaRot first part: dx/dy is inverse of dy/dx
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-28T17:24:10.283Z
Answers.Text: It sounds like the maximum gradient constraint is enforced in the forward() function of the discriminator.
Answers.UserDisplayName: russian_spy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-30T15:05:11.510Z
Answers.LastActivityDate: 2021-08-31T08:06:37.873Z
Answers.LastEditDate: 2021-08-31T08:06:37.873Z
Answers.UserDisplayName: Criticize SE actions means ban
Answers.Body: <p>Optimizing a traditional GAN is equivalent to minimizing KL-divergence, which is known to have limitations. Wasserstein metric promises to remedy these limitations, and is defined as follows:</p>
  <p><span class="math-container">$$
  W(P_r, P_g) = \inf_{\gamma \in \prod(P_r ,P_g)} \mathbb{E}_{(x, y) \sim \gamma}\big[\:\|x - y\|\:\big]
  $$</span></p>
  <p>However, computing <span class="math-container">$\prod(P_r, P_g)$</span> for all possible joint probability distributions is extremely intractable. Thus the authors proposed a smart transformation of the formula based on the <a href="https://en.wikipedia.org/wiki/Wasserstein_metric#Dual_representation_of_W1" rel="nofollow noreferrer">Kantorovich-Rubinstein duality</a>, defined as:</p>
  <p><span class="math-container">$$
  W(P_r, P_\theta) = \sup_{\|f\|_L \leq K}
  \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{x \sim P_\theta}[f(x)]
  $$</span></p>
  <p>For this to work, <span class="math-container">$\| f \|_L \leq K$</span> must hold, which means that <span class="math-container">$f$</span> must be K-Lipschitz continuous. <span class="math-container">$f$</span> is called K-Lipschitz continuous if there exists a real constant <span class="math-container">$K \geq 0$</span> such that, for all <span class="math-container">$x_1, x_2 \in \mathbb{R}$</span>,</p>
  <p><span class="math-container">$$\lvert f(x_1) - f(x_2) \rvert \leq K \lvert x_1 - x_2 \rvert$$</span> ​
  Here <span class="math-container">$K$</span> is known as a Lipschitz constant for the function <span class="math-container">$f$</span>. <a href="https://en.wikipedia.org/wiki/Lipschitz_continuity" rel="nofollow noreferrer">Lipschitz continuity</a> ensures that the derivative of <span class="math-container">$f$</span> is less than or equal to K everywhere (or to 1 for 1-Lipschitz). This can be illustrated as follows:</p>
  <p><a href="https://upload.wikimedia.org/wikipedia/commons/5/58/Lipschitz_Visualisierung.gif" rel="nofollow noreferrer"><img src="https://upload.wikimedia.org/wikipedia/commons/5/58/Lipschitz_Visualisierung.gif" alt="enter image description here" /></a></p>
  <blockquote>
  <p>For a Lipschitz continuous function, there exists a double cone
  (white) whose origin can be moved along the graph so that the whole
  graph always stays outside the double cone</p>
  </blockquote>
  <p>Thus, we have a parametrized family of functions <span class="math-container">$\{f_w\}_{w \in W}$</span> that are all K-Lipschitz for some <span class="math-container">$K$</span>. The function <span class="math-container">$f$</span> has the role of a non-linear feature map that maximally enhances the differences between the samples coming from the two distributions. The role of the Lipschitz constraint is to prevent <span class="math-container">$f$</span> from arbitrarily enhancing small differences. The constraint assures that if two input images are similar the output of <span class="math-container">$f$</span> will be similar as well. Without this constraint, the result would be zero when <span class="math-container">$P_r$</span> is equal to <span class="math-container">$P_g$</span> and <span class="math-container">$\infty$</span> otherwise, since the effect of any minor difference can be arbitrarily enhanced by an appropriate feature map.</p>
  <p>The supremum over <span class="math-container">$K$</span>-Lipschitz functions is still intractable, but now <span class="math-container">$\{f_w\}$</span> can be approximated using a neural network. For optimization purposes, it is not necessary to know what <span class="math-container">$K$</span> is. It is enough to know that it exists and that it is fixed throughout the training process (read this <a href="https://www.alexirpan.com/2017/02/22/wasserstein-gan.html" rel="nofollow noreferrer">article</a> for more details).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-31T14:36:44.403Z
Answers.LastActivityDate: 2021-07-31T14:36:44.403Z
Answers.UserDisplayName: Aray Karjauv
Answers.Body: <p>Lipschitz continuity is a mathematical property that measures how fast a function can change as its input changes.
  In the context of GANs, it is desirable to enforce Lipschitz continuity on the discriminator function to ensure that it does not change too quickly and cause instability during training.</p>
  <p>Critic’s neural network needs to be 1-L(1-Lipschitz) Continuous when using W-Loss, which means that the norm of its gradient can't be greater than one. All points on the function are considered for the evaluation of 1-Lipschitz continuity.</p>
  <p>Intuitively, this means that a Lipschitz continuous function can never vary too rapidly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-07T07:27:24.950Z
Answers.LastActivityDate: 2023-05-07T07:27:24.950Z
Answers.UserDisplayName: Naren Babu R
Body: <p>The following is the abstract for the research paper titled <a href="https://arxiv.org/pdf/1704.00028v3.pdf" rel="noreferrer">Improved Training of Wasserstein GANs</a></p>
  <blockquote>
  <p>Generative Adversarial Networks (GANs) are powerful generative models,
  but suffer from training instability. The recently proposed
  Wasserstein GAN (WGAN) makes progress toward stable training of GANs,
  but sometimes can still generate only poor samples or fail to
  converge. We find that <strong>these problems are often due to the use of
  weight clipping in WGAN to enforce a Lipschitz constraint on the
  critic, which can lead to undesired behavior.</strong> We propose an
  alternative to clipping weights: penalize the norm of gradient of the
  critic with respect to its input. Our proposed method performs better
  than standard WGAN and enables stable training of a wide variety of
  GAN architectures with almost no hyperparameter tuning, including
  101-layer ResNets and language models with continuous generators. We
  also achieve high quality generations on CIFAR-10 and LSUN bedrooms.</p>
  </blockquote>
  <p>Here, the critic stands for discriminator of the GAN. I understood that the discriminator must obey <strong>Lipschitz constraint</strong> and hence weight clipping is generally done before this paper. The paper provides an alternative way, penalizing the norm of the gradient of the critic with respect to its input, to enforce the desired Lipschitz constraint.</p>
  <p>What actually is Lipschitz constraint and why is it mandatory for a discriminator to obey it?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-30T10:29:26.527Z
Comments.Text: Perhaps follow the reference to the WGAN paper and read about it?
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-30T10:31:23.413Z
Comments.Text: @user253751 It's in my pipeline, if anyone got an easy interpretation, then it will be useful since it seems to be mathematical.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-22T06:24:29.457Z
Comments.Text: [p:90](https://www.deeplearningbook.org/contents/numerical.html) of deep learning book
Comments.UserDisplayName: hanugm
Tags: <training><math><generative-adversarial-networks><discriminator>
Title: What is Lipschitz constraint and why it is enforced on discriminator?
UserDisplayName: hanugm
ThreadText: # What is Lipschitz constraint and why it is enforced on discriminator?
  
  (Asked by: username_0 on 2021-07-29)
  
  The following is the abstract for the research paper titled [Improved Training of Wasserstein GANs](https://arxiv.org/pdf/1704.00028v3.pdf)
  
  > Generative Adversarial Networks (GANs) are powerful generative models,
  > but suffer from training instability. The recently proposed
  > Wasserstein GAN (WGAN) makes progress toward stable training of GANs,
  > but sometimes can still generate only poor samples or fail to
  > converge. We find that **these problems are often due to the use of
  > weight clipping in WGAN to enforce a Lipschitz constraint on the
  > critic, which can lead to undesired behavior.** We propose an
  > alternative to clipping weights: penalize the norm of gradient of the
  > critic with respect to its input. Our proposed method performs better
  > than standard WGAN and enables stable training of a wide variety of
  > GAN architectures with almost no hyperparameter tuning, including
  > 101-layer ResNets and language models with continuous generators. We
  > also achieve high quality generations on CIFAR-10 and LSUN bedrooms.
  
  Here, the critic stands for discriminator of the GAN. I understood that the discriminator must obey **Lipschitz constraint** and hence weight clipping is generally done before this paper. The paper provides an alternative way, penalizing the norm of the gradient of the critic with respect to its input, to enforce the desired Lipschitz constraint.
  
  What actually is Lipschitz constraint and why is it mandatory for a discriminator to obey it?
  
  **Tags:** training, math, generative-adversarial-networks, discriminator
  > **Comments:**
  > **username_1**: Perhaps follow the reference to the WGAN paper and read about it?  
  > **username_0**: @user253751 It's in my pipeline, if anyone got an easy interpretation, then it will be useful since it seems to be mathematical.  
  > **username_0**: [p:90](https://www.deeplearningbook.org/contents/numerical.html) of deep learning book  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2021-07-30. Score: 7
  
  The [Lipschitz constraint](https://en.wikipedia.org/wiki/Lipschitz_continuity) is essentially that a function must have a maximum gradient. The specific maximum gradient is a hyperparameter.
  
  It's **not** mandatory for a discriminator to obey a Lipschitz constraint. However, in [the WGAN paper](https://arxiv.org/pdf/1701.07875.pdf) they find that *if* the discriminator *does* obey a Lipschitz constraint, the GAN works much better.
  
  A perfect discriminator would perfectly accept all the real samples (output=1 with low gradient) and perfectly reject all the fake samples (output=0 with low gradient), but this provides hardly any gradient information to help train the generator. By limiting the gradient of the discriminator, they force it to be a worse discriminator, but provide more gradient information which helps train the generator.
  
  You can see this in figure 2 (page 9) of the WGAN paper. The red line is a good discriminator but its gradient is nearly 0 at most points. The cyan line (which for some reason is presented upside-down) is clearly much worse as a discriminator, but is much better for training the generator because its gradient is not zero.
  
  The way they limit the discriminator's *overall* gradient in the WGAN paper is by separately limiting (clipping) each of the weights in the discriminator. They are aware this is a "terrible" idea, and leave better ideas for further research. The paper you are asking about is one of those better ideas.
  
  Note that since back-propagation works backwards, enforcing a *maximum* gradient (Lipschitz continuity) on the discriminator in the *forward* direction causes it to have a *minimum* gradient in the *backward* direction, which is what we want when training the generator.
  
  > **Comments:**
  > **username_2**: can you explain more about why enforcing a maximum gradient on the discriminator in forwarding leads to the minimum gradient in the backward direction in generator? And why why small gradient is good for generator?  
  > **username_1**: @CuCaRot first part: dx/dy is inverse of dy/dx  
  > **username_3**: It sounds like the maximum gradient constraint is enforced in the forward() function of the discriminator.  
  
  ---
  
  ### Answer by username_4 on 2021-07-31. Score: 4
  
  Optimizing a traditional GAN is equivalent to minimizing KL-divergence, which is known to have limitations. Wasserstein metric promises to remedy these limitations, and is defined as follows:
  
  $$
  W(P\_r, P\_g) = \inf\_{\gamma \in \prod(P\_r ,P\_g)} \mathbb{E}\_{(x, y) \sim \gamma}\big[\:\|x - y\|\:\big]
  $$
  
  However, computing $\prod(P\_r, P\_g)$ for all possible joint probability distributions is extremely intractable. Thus the authors proposed a smart transformation of the formula based on the [Kantorovich-Rubinstein duality](https://en.wikipedia.org/wiki/Wasserstein_metric#Dual_representation_of_W1), defined as:
  
  $$
  W(P\_r, P\_\theta) = \sup\_{\|f\|\_L \leq K}
  \mathbb{E}\_{x \sim P\_r}[f(x)] - \mathbb{E}\_{x \sim P\_\theta}[f(x)]
  $$
  
  For this to work, $\| f \|\_L \leq K$ must hold, which means that $f$ must be K-Lipschitz continuous. $f$ is called K-Lipschitz continuous if there exists a real constant $K \geq 0$ such that, for all $x\_1, x\_2 \in \mathbb{R}$,
  
  $$\lvert f(x\_1) - f(x\_2) \rvert \leq K \lvert x\_1 - x\_2 \rvert$$ ​
  Here $K$ is known as a Lipschitz constant for the function $f$. [Lipschitz continuity](https://en.wikipedia.org/wiki/Lipschitz_continuity) ensures that the derivative of $f$ is less than or equal to K everywhere (or to 1 for 1-Lipschitz). This can be illustrated as follows:
  
  [![enter image description here](https://upload.wikimedia.org/wikipedia/commons/5/58/Lipschitz_Visualisierung.gif)](https://upload.wikimedia.org/wikipedia/commons/5/58/Lipschitz_Visualisierung.gif)
  
  > For a Lipschitz continuous function, there exists a double cone
  > (white) whose origin can be moved along the graph so that the whole
  > graph always stays outside the double cone
  
  Thus, we have a parametrized family of functions $\{f\_w\}\_{w \in W}$ that are all K-Lipschitz for some $K$. The function $f$ has the role of a non-linear feature map that maximally enhances the differences between the samples coming from the two distributions. The role of the Lipschitz constraint is to prevent $f$ from arbitrarily enhancing small differences. The constraint assures that if two input images are similar the output of $f$ will be similar as well. Without this constraint, the result would be zero when $P\_r$ is equal to $P\_g$ and $\infty$ otherwise, since the effect of any minor difference can be arbitrarily enhanced by an appropriate feature map.
  
  The supremum over $K$-Lipschitz functions is still intractable, but now $\{f\_w\}$ can be approximated using a neural network. For optimization purposes, it is not necessary to know what $K$ is. It is enough to know that it exists and that it is fixed throughout the training process (read this [article](https://www.alexirpan.com/2017/02/22/wasserstein-gan.html) for more details).
  
  
  ---
  
  ### Answer by username_5 on 2023-05-07. Score: 0
  
  Lipschitz continuity is a mathematical property that measures how fast a function can change as its input changes.
  In the context of GANs, it is desirable to enforce Lipschitz continuity on the discriminator function to ensure that it does not change too quickly and cause instability during training.
  
  Critic’s neural network needs to be 1-L(1-Lipschitz) Continuous when using W-Loss, which means that the norm of its gradient can't be greater than one. All points on the function are considered for the evaluation of 1-Lipschitz continuity.
  
  Intuitively, this means that a Lipschitz continuous function can never vary too rapidly.
  
  
  ---
  
  