Answers.Body: <p>First of all, the discriminator in WGAN does not give a value in the range <span class="math-container">$[0,1]$</span>. Compared to the traditional discriminator, it has a linear activation in the output layer. Therefore, the authors call it critic instead.</p>
  <p>To calculate the penalty, we sample an image that lies on the line between the real and the generated image. This is done by sampling a real image <span class="math-container">$x$</span>, generating an image <span class="math-container">$\tilde{x}$</span>, and mixing these images <span class="math-container">$\hat{x} = \alpha \widetilde{x}+(1-\alpha)x$</span> with <span class="math-container">$\alpha \sim U(0,1)$</span>. That is, <span class="math-container">$\hat{x}$</span> is uniformly sampled from the lines between real and fake images, which can be illustrated as follows:</p>
  <p><a href="https://i.sstatic.net/hKmfk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hKmfk.png" alt="enter image description here" /></a></p>
  <p>We then feed <span class="math-container">$\hat{x}$</span> into the critic and calculate the gradient norm of the discriminator's output with respect to its input, <span class="math-container">$(\| \triangledown_{\hat{x}} D(\hat{x})\|_2 - 1 )^2$</span>. Here is a snippet code for PyTorch:</p>
  <pre><code>def gradient_penalty(D, real_data, generated_data, device):
      batch_size = real_data.shape[0]
  
      # Calculate interpolation
      alpha = torch.rand(batch_size, 1, 1, 1)
      alpha = alpha.expand_as(real_data).to(device)
      # getting x hat
      interpolated = alpha * real_data + (1 - alpha) * generated_data
  
      dis_interpolated = D(interpolated)
      grad_outputs = torch.ones(dis_interpolated.shape).to(device)
  
      # Calculate gradients of probabilities with respect to examples
      gradients = autograd.grad(outputs=dis_interpolated, inputs=interpolated,
                             grad_outputs=grad_outputs, create_graph=True, retain_graph=True)[0]
  
      # Gradients have shape (batch_size, num_channels, img_width, img_height),
      # so flatten to easily take norm per example in batch
      gradients = gradients.view(batch_size, -1)
  
      # Derivatives of the gradient close to 0 can cause problems because of
      # the square root, so manually calculate norm and add epsilon
      gradients_norm = ((torch.sqrt(torch.sum(gradients ** 2, dim=1) + 1e-12) - 1) ** 2).mean()
      return gradients_norm
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-30T02:19:13.583Z
Answers.LastActivityDate: 2021-07-30T02:19:13.583Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>The research paper titled <a href="https://arxiv.org/pdf/1704.00028.pdf" rel="nofollow noreferrer">Improved Training of Wasserstein GANs</a> proposed a gradient penalty in order to avoid undesired behavior due to weight clipping of the discriminator.</p>
  <blockquote>
  <p>We now propose an alternative way to enforce the Lipschitz constraint.
  A differentiable function is 1-Lipschtiz if and only if it has
  gradients with norm at most 1 everywhere, so
  we consider directly constraining the gradient norm of the critic’s
  output with respect to its input. To circumvent tractability issues,
  we enforce a soft version of the constraint with a penalty on the
  gradient norm for random samples <span class="math-container">$\hat{x} \sim  P_\hat{x}$</span>. Our new
  objective is</p>
  <p><span class="math-container">$$L = \mathop{\mathbb{E}}\limits_{\tilde x \sim \mathbb{P}_g}
   [D(\tilde x)] - \mathop{\mathbb{E}}\limits_{ x \sim \mathbb{P}_r}
   [D(x)] +  \mathop{\mathbb{E}}_{\hat{x}  \sim P_{\hat{x}}} [
    (\| \triangledown_{\hat{x}} D(\hat{x})\|_2 - 1 )^2  ]$$</span></p>
  </blockquote>
  <p>The last term in the discriminator's loss function is related to the gradient penalty. It is easy to calculate the first two terms. Since discriminator, in general, gives value in range <span class="math-container">$[0, 1]$</span>, the first two terms are just the average of the sequence of probability values given by discriminator on generated and real images respectively.</p>
  <p>But, how to calculate <span class="math-container">$\triangledown_{\hat{x}} D(\hat{x})$</span> for a given image <span class="math-container">$\hat{x}$</span>?</p>
  
Tags: <training><math><implementation><gradient><wasserstein-gan>
Title: How to calculate the gradient penalty proposed in "Improved Training of Wasserstein GANs"?
UserDisplayName: hanugm
ThreadText: # How to calculate the gradient penalty proposed in "Improved Training of Wasserstein GANs"?
  
  (Asked by: username_0 on 2021-07-30)
  
  The research paper titled [Improved Training of Wasserstein GANs](https://arxiv.org/pdf/1704.00028.pdf) proposed a gradient penalty in order to avoid undesired behavior due to weight clipping of the discriminator.
  
  > We now propose an alternative way to enforce the Lipschitz constraint.
  > A differentiable function is 1-Lipschtiz if and only if it has
  > gradients with norm at most 1 everywhere, so
  > we consider directly constraining the gradient norm of the critic’s
  > output with respect to its input. To circumvent tractability issues,
  > we enforce a soft version of the constraint with a penalty on the
  > gradient norm for random samples $\hat{x} \sim P\_\hat{x}$. Our new
  > objective is
  >
  > $$L = \mathop{\mathbb{E}}\limits\_{\tilde x \sim \mathbb{P}\_g}
  > [D(\tilde x)] - \mathop{\mathbb{E}}\limits\_{ x \sim \mathbb{P}\_r}
  > [D(x)] + \mathop{\mathbb{E}}\_{\hat{x} \sim P\_{\hat{x}}} [
  > (\| \triangledown\_{\hat{x}} D(\hat{x})\|\_2 - 1 )^2 ]$$
  
  The last term in the discriminator's loss function is related to the gradient penalty. It is easy to calculate the first two terms. Since discriminator, in general, gives value in range $[0, 1]$, the first two terms are just the average of the sequence of probability values given by discriminator on generated and real images respectively.
  
  But, how to calculate $\triangledown\_{\hat{x}} D(\hat{x})$ for a given image $\hat{x}$?
  
  **Tags:** training, math, implementation, gradient, wasserstein-gan
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-30. Score: 2
  
  First of all, the discriminator in WGAN does not give a value in the range $[0,1]$. Compared to the traditional discriminator, it has a linear activation in the output layer. Therefore, the authors call it critic instead.
  To calculate the penalty, we sample an image that lies on the line between the real and the generated image. This is done by sampling a real image $x$, generating an image $\tilde{x}$, and mixing these images $\hat{x} = \alpha \widetilde{x}+(1-\alpha)x$ with $\alpha \sim U(0,1)$. That is, $\hat{x}$ is uniformly sampled from the lines between real and fake images, which can be illustrated as follows:
  
  We then feed $\hat{x}$ into the critic and calculate the gradient norm of the discriminator's output with respect to its input, $(\| \triangledown_{\hat{x}} D(\hat{x})\|_2 - 1 )^2$. Here is a snippet code for PyTorch:
  def gradient_penalty(D, real_data, generated_data, device):
      batch_size = real_data.shape[0]
  
      # Calculate interpolation
      alpha = torch.rand(batch_size, 1, 1, 1)
      alpha = alpha.expand_as(real_data).to(device)
      # getting x hat
      interpolated = alpha * real_data + (1 - alpha) * generated_data
  
      dis_interpolated = D(interpolated)
      grad_outputs = torch.ones(dis_interpolated.shape).to(device)
  
      # Calculate gradients of probabilities with respect to examples
      gradients = autograd.grad(outputs=dis_interpolated, inputs=interpolated,
                             grad_outputs=grad_outputs, create_graph=True, retain_graph=True)[0]
  
      # Gradients have shape (batch_size, num_channels, img_width, img_height),
      # so flatten to easily take norm per example in batch
      gradients = gradients.view(batch_size, -1)
  
      # Derivatives of the gradient close to 0 can cause problems because of
      # the square root, so manually calculate norm and add epsilon
      gradients_norm = ((torch.sqrt(torch.sum(gradients ** 2, dim=1) + 1e-12) - 1) ** 2).mean()
      return gradients_norm
  
  
  ---
  
  