Answers.Body: <p>In [1], section 4, the authors mention the following:</p>
  <blockquote>
  <p><strong>Sampling distribution</strong> We implicitly define <span class="math-container">$p_{\hat{x}}$</span> sampling uniformly along straight lines between pairs of points sampled from the data distribution <span class="math-container">$p_{data}$</span> and the generator distribution <span class="math-container">$p_{G}$</span>. This is motivated by the fact that the optimal critic contains straight lines with gradient norm 1 connecting coupled points from <span class="math-container">$p_{data}$</span> and <span class="math-container">$p_{G}$</span> (see Proposition 1). <strong>Given that enforcing the unit gradient norm constraint everywhere is intractable</strong> (1), <strong>enforcing it only along these straight lines seems sufficient and experimentally results in good performance</strong> (2).</p>
  </blockquote>
  <p>So, there are 2 reasons for enforcing <span class="math-container">$\nabla f_{\phi} = 1$</span> over <span class="math-container">$\hat{x} = tx + (1 - t) G(z)$</span>. Starting with point (1),</p>
  <p>(1) Enforcing <span class="math-container">$\nabla f_{\phi} = 1$</span> everywhere would amount to sampling <span class="math-container">$x$</span> from the whole space <span class="math-container">$\mathbb{R}^{d}$</span> uniformly. By the curse of dimensionality, this is unfeasible.</p>
  <p>Now, let us consider why does it suffice to enforce on samples from <span class="math-container">$p_{\hat{x}}$</span>. Note that, due to <em>displacement interpolation</em> [2, Theorem 7.21], the interpolation of two probability measures under the Wasserstein metric corresponds to following geodesics in the ambient space.</p>
  <p>In the case of GANs, let <span class="math-container">$x \sim p_{data}$</span> and <span class="math-container">$\tilde{x} \sim p_{model}$</span>, where <span class="math-container">$p_{model} = G_{\sharp}p_{G}$</span>. The interpolation <span class="math-container">$p_{t}$</span> of these two distributions is supported on points <span class="math-container">$\hat{x}_{t}=tx + (1-t)\tilde{x}$</span>, that is, the distribution <span class="math-container">$p_{\hat{x}}$</span>, for fixed <span class="math-container">$t$</span>, is actually the interpolation in probability space of <span class="math-container">$p_{data}$</span> and <span class="math-container">$p_{model}$</span>. As consequence, the authors propose enforcing Lipschitzness on a much more reduced region of <span class="math-container">$\mathbb{R}^{d}$</span>, that is, the support of <span class="math-container">$p_{data}$</span>, <span class="math-container">$p_{model}$</span> and interpolations <span class="math-container">$p_{t}$</span>. This <em>may be justified</em> (although the authors do not give theoretical reasons) via the <a href="https://en.wikipedia.org/wiki/Manifold_hypothesis" rel="nofollow noreferrer">Manifold Hypothesis</a>, that is, the data is supposed to lie on a non-linear low-dimensional region of <span class="math-container">$\mathbb{R}^{d}$</span> (so that we do not need to regularize over the whole space).</p>
  <p><strong>Remark:</strong> the book by Villani [2] is rather technical, and for a reader not familiarized with optimal transport it may be challenging to understand it. A somewhat easier text is the book by Santambrogio [3], which discusses displacement interpolation under the Wasserstein metric in chapter 5. All in all I consider that the authors had an interesting theoretical intuition, and that they validated their approach empirically, but the question on <strong>why</strong> does this work is not very clear.</p>
  <h2>References</h2>
  <p>[1] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., &amp; Courville, A. C. (2017). <a href="https://arxiv.org/pdf/1704.00028.pdf" rel="nofollow noreferrer">Improved training of wasserstein gans</a>. Advances in neural information processing systems, 30.</p>
  <p>[2] Villani, C. (2009). Optimal transport: old and new (Vol. 338, p. 23). Berlin: springer.</p>
  <p>[3] Santambrogio, F. (2015). Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63), 94.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-31T11:11:54.623Z
Answers.LastActivityDate: 2022-05-31T11:11:54.623Z
Answers.UserDisplayName: Eduardo Montesuma
Body: <p>I'm trying to better frame/summarize the formulations and motivations behind <em>Wasserstein GAN</em> with gradient penalty, based on my understanding.</p>
  <p>For the basic GAN we are trying to optimize the following quantity:</p>
  <p><span class="math-container">$$\min_\theta \max_\phi \mathbb{E}_{x \sim p_{data}(x)}[D_\phi(x)] + \mathbb{E}_{z \sim p_G(z)}[1-D(G_\theta(z))]$$</span></p>
  <p>The problem is that the dissimilarity measure between the two probabilities given by <em>Jensen-Shannon divergence</em> will not take into account any <strong>distance</strong> in a <em>Euclidean</em> sense. That's why we consider the Wasserstein distance defined as:</p>
  <p><span class="math-container">$$W(p_{data}, p_G) := \inf_\gamma \, \,\mathbb{E}_{(x,y) \sim \gamma(x,y)}\|x-y\|$$</span></p>
  <p>that will account for a proper distance of our distributions. Computing it is very hard so we rely on Kantorovich-Rubinstein duality which states we can rewrite <span class="math-container">$W$</span> as:</p>
  <p><span class="math-container">$$W(p_{data},p_G) = \sup_{\|f\|_L \le 1}\mathbb{E}_{x \sim p_{data}(x)}[f_\phi(x)] - \mathbb{E}_{z \sim p_{G}(x)}[f_\phi(G_\theta(z))]$$</span></p>
  <p>Now the crucial point, to enforce the constraint of <span class="math-container">$1$</span>-Lipschitz continuity of the discriminator we add a penalty term to bound the norm of the gradient of <span class="math-container">$f$</span>, so the final loss we consider is:</p>
  <p><span class="math-container">$$\mathcal{L} = \mathbb{E}_{x \sim p_{data}(x)}[f_\phi(x)] - \mathbb{E}_{z \sim p_{G}(x)}[f_\phi(G_\theta(z))] + \lambda \, \mathbb{E}_{\hat{x}}[(\|\nabla_{\hat{x}} f_\theta(\hat{x})\|-1)^2]$$</span></p>
  <p>where</p>
  <p><span class="math-container">\begin{equation}
  \hat{x} = tx + (1-t)z
  \end{equation}</span>
  <span class="math-container">$t \in [0,1]$</span>.</p>
  <p>Now, I've understood that we bound the slope of discriminator because we want toavoid the vanishing gradient problem and keep gradient signal in order to make the generator learn, but why do we actually penalize the gradient of discriminator with respect to a linear interpolation of real and fake data?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-30T19:26:57.840Z
Comments.Text: Added bounty because it's a really interesting question. GANs made the waves they did because the discriminator they could quantify that the generator was getting "closer" in a more sophisticated way than just measuring pixelwise distance. It does seem quite strange that the gradient penalty would just be done with some pixelwise mixture.
Comments.UserDisplayName: Josiah
Tags: <generative-adversarial-networks><wasserstein-gan><wasserstein-metric>
Title: Why do we use a linear interpolation of fake and real data to penalize the gradient of discriminator in WGAN-GP
UserDisplayName: James Arten
ThreadText: # Why do we use a linear interpolation of fake and real data to penalize the gradient of discriminator in WGAN-GP
  
  (Asked by: username_0 on 2022-03-22)
  
  I'm trying to better frame/summarize the formulations and motivations behind *Wasserstein GAN* with gradient penalty, based on my understanding.
  
  For the basic GAN we are trying to optimize the following quantity:
  
  $$\min\_\theta \max\_\phi \mathbb{E}\_{x \sim p\_{data}(x)}[D\_\phi(x)] + \mathbb{E}\_{z \sim p\_G(z)}[1-D(G\_\theta(z))]$$
  
  The problem is that the dissimilarity measure between the two probabilities given by *Jensen-Shannon divergence* will not take into account any **distance** in a *Euclidean* sense. That's why we consider the Wasserstein distance defined as:
  
  $$W(p\_{data}, p\_G) := \inf\_\gamma \, \,\mathbb{E}\_{(x,y) \sim \gamma(x,y)}\|x-y\|$$
  
  that will account for a proper distance of our distributions. Computing it is very hard so we rely on Kantorovich-Rubinstein duality which states we can rewrite $W$ as:
  
  $$W(p\_{data},p\_G) = \sup\_{\|f\|\_L \le 1}\mathbb{E}\_{x \sim p\_{data}(x)}[f\_\phi(x)] - \mathbb{E}\_{z \sim p\_{G}(x)}[f\_\phi(G\_\theta(z))]$$
  
  Now the crucial point, to enforce the constraint of $1$-Lipschitz continuity of the discriminator we add a penalty term to bound the norm of the gradient of $f$, so the final loss we consider is:
  
  $$\mathcal{L} = \mathbb{E}\_{x \sim p\_{data}(x)}[f\_\phi(x)] - \mathbb{E}\_{z \sim p\_{G}(x)}[f\_\phi(G\_\theta(z))] + \lambda \, \mathbb{E}\_{\hat{x}}[(\|\nabla\_{\hat{x}} f\_\theta(\hat{x})\|-1)^2]$$
  
  where
  
  \begin{equation}
  \hat{x} = tx + (1-t)z
  \end{equation}
  $t \in [0,1]$.
  
  Now, I've understood that we bound the slope of discriminator because we want toavoid the vanishing gradient problem and keep gradient signal in order to make the generator learn, but why do we actually penalize the gradient of discriminator with respect to a linear interpolation of real and fake data?
  
  **Tags:** generative-adversarial-networks, wasserstein-gan, wasserstein-metric
  
  > **Comments:**
  > **username_1**: Added bounty because it's a really interesting question. GANs made the waves they did because the discriminator they could quantify that the generator was getting "closer" in a more sophisticated way than just measuring pixelwise distance. It does seem quite strange that the gradient penalty would just be done with some pixelwise mixture.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-05-31. Score: 1
  
  In [1], section 4, the authors mention the following:
  
  > **Sampling distribution** We implicitly define $p\_{\hat{x}}$ sampling uniformly along straight lines between pairs of points sampled from the data distribution $p\_{data}$ and the generator distribution $p\_{G}$. This is motivated by the fact that the optimal critic contains straight lines with gradient norm 1 connecting coupled points from $p\_{data}$ and $p\_{G}$ (see Proposition 1). **Given that enforcing the unit gradient norm constraint everywhere is intractable** (1), **enforcing it only along these straight lines seems sufficient and experimentally results in good performance** (2).
  
  So, there are 2 reasons for enforcing $\nabla f\_{\phi} = 1$ over $\hat{x} = tx + (1 - t) G(z)$. Starting with point (1),
  
  (1) Enforcing $\nabla f\_{\phi} = 1$ everywhere would amount to sampling $x$ from the whole space $\mathbb{R}^{d}$ uniformly. By the curse of dimensionality, this is unfeasible.
  
  Now, let us consider why does it suffice to enforce on samples from $p\_{\hat{x}}$. Note that, due to *displacement interpolation* [2, Theorem 7.21], the interpolation of two probability measures under the Wasserstein metric corresponds to following geodesics in the ambient space.
  
  In the case of GANs, let $x \sim p\_{data}$ and $\tilde{x} \sim p\_{model}$, where $p\_{model} = G\_{\sharp}p\_{G}$. The interpolation $p\_{t}$ of these two distributions is supported on points $\hat{x}\_{t}=tx + (1-t)\tilde{x}$, that is, the distribution $p\_{\hat{x}}$, for fixed $t$, is actually the interpolation in probability space of $p\_{data}$ and $p\_{model}$. As consequence, the authors propose enforcing Lipschitzness on a much more reduced region of $\mathbb{R}^{d}$, that is, the support of $p\_{data}$, $p\_{model}$ and interpolations $p\_{t}$. This *may be justified* (although the authors do not give theoretical reasons) via the [Manifold Hypothesis](https://en.wikipedia.org/wiki/Manifold_hypothesis), that is, the data is supposed to lie on a non-linear low-dimensional region of $\mathbb{R}^{d}$ (so that we do not need to regularize over the whole space).
  
  **Remark:** the book by Villani [2] is rather technical, and for a reader not familiarized with optimal transport it may be challenging to understand it. A somewhat easier text is the book by Santambrogio [3], which discusses displacement interpolation under the Wasserstein metric in chapter 5. All in all I consider that the authors had an interesting theoretical intuition, and that they validated their approach empirically, but the question on **why** does this work is not very clear.
  
  ## References
  
  [1] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. C. (2017). [Improved training of wasserstein gans](https://arxiv.org/pdf/1704.00028.pdf). Advances in neural information processing systems, 30.
  
  [2] Villani, C. (2009). Optimal transport: old and new (Vol. 338, p. 23). Berlin: springer.
  
  [3] Santambrogio, F. (2015). Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63), 94.
  
  
  ---
  
  