Answers.Body: <p>The terms &quot;<em>insufficient gradient</em>&quot; or &quot;<em>not strong enough gradient</em>&quot; usually means that the magnitude of the gradient vector is too small or nearly zero that they can't drive the optimization properly.</p>
  <p>Not having sufficient gradient is similar to having a very low learning rate - they are not only slow (in terms of convergence) but also drifts the optimization in a poor direction and get stuck in local minima.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-09T03:18:43.883Z
Answers.LastActivityDate: 2021-08-09T14:07:51.833Z
Answers.LastEditDate: 2021-08-09T14:07:51.833Z
Answers.UserDisplayName: ayandas
Body: <p>It has been mentioned in the research paper titled <a href="https://papers.nips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf" rel="nofollow noreferrer">Generative Adversarial Nets</a> that generator need to maximize the function <span class="math-container">$\log D(G(z))$</span> instead of minimizing  <span class="math-container">$\log(1 −D(G(z)))$</span> since the former provides sufficient gradient than latter.</p>
  <blockquote>
  <p><span class="math-container">$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] + 
   \mathbb{E}_{z ∼ p_z}[log (1 - D(z))]$$</span>
  In practice, the above equation <strong>may not provide sufficient gradient
  for <span class="math-container">$G$</span> to learn well</strong>. Early in learning, when <span class="math-container">$G$</span> is poor, <span class="math-container">$D$</span> can
  reject samples with high confidence because they are clearly different
  from the training data. In this case, <span class="math-container">$\log(1 −D(G(z)))$</span> saturates.
  Rather than training <span class="math-container">$G$</span> to minimize <span class="math-container">$\log(1 −D(G(z)))$</span> we can train G
  to maximize <span class="math-container">$\log D(G(z))$</span>. This objective function results in the same
  fixed point of the dynamics of <span class="math-container">$G$</span> and <span class="math-container">$D$</span> but <strong>provides much stronger
  gradients</strong> early in learning.</p>
  </blockquote>
  <p>A gradient is a vector containing the partial derivatives of outputs w.r.t inputs. At a particular point, the gradient is a vector of real numbers. These gradients are useful in the training phase by providing direction-related information and the magnitude of step in the opposite direction. This is my understanding regarding gradients.</p>
  <p>What is meant by sufficient or strong gradient? Is it the norm of the gradient or some other measure on the gradient vector?</p>
  <p>If possible, please show an example of strong and weak gradients with numbers so that I can quickly understand.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-09T02:56:43.563Z
Comments.Text: Technically, the term "sufficient graident" isn't quite well-defined. But usually what it means is the magnitude of the gradients are so small that it is unable to drive the optimization process.
Comments.UserDisplayName: ayandas
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-09T02:58:22.500Z
Comments.Text: @AyanDas Does it mean they update weights very slowly or do not converge?
Comments.UserDisplayName: hanugm
Tags: <training><terminology><papers><generative-adversarial-networks><gradient>
Title: What does it mean by strong or sufficient gradient for training in this context?
UserDisplayName: hanugm
ThreadText: # What does it mean by strong or sufficient gradient for training in this context?
  
  (Asked by: username_0 on 2021-08-03)
  
  It has been mentioned in the research paper titled [Generative Adversarial Nets](https://papers.nips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf) that generator need to maximize the function $\log D(G(z))$ instead of minimizing $\log(1 −D(G(z)))$ since the former provides sufficient gradient than latter.
  
  > $$\min\_G \max\_DV(D, G) = \mathbb{E}\_{x ∼ P\_{data}}[\log D(x)] +
  > \mathbb{E}\_{z ∼ p\_z}[log (1 - D(z))]$$
  > In practice, the above equation **may not provide sufficient gradient
  > for $G$ to learn well**. Early in learning, when $G$ is poor, $D$ can
  > reject samples with high confidence because they are clearly different
  > from the training data. In this case, $\log(1 −D(G(z)))$ saturates.
  > Rather than training $G$ to minimize $\log(1 −D(G(z)))$ we can train G
  > to maximize $\log D(G(z))$. This objective function results in the same
  > fixed point of the dynamics of $G$ and $D$ but **provides much stronger
  > gradients** early in learning.
  
  A gradient is a vector containing the partial derivatives of outputs w.r.t inputs. At a particular point, the gradient is a vector of real numbers. These gradients are useful in the training phase by providing direction-related information and the magnitude of step in the opposite direction. This is my understanding regarding gradients.
  
  What is meant by sufficient or strong gradient? Is it the norm of the gradient or some other measure on the gradient vector?
  
  If possible, please show an example of strong and weak gradients with numbers so that I can quickly understand.
  
  **Tags:** training, terminology, papers, generative-adversarial-networks, gradient
  > **Comments:**
  > **username_1**: Technically, the term "sufficient graident" isn't quite well-defined. But usually what it means is the magnitude of the gradients are so small that it is unable to drive the optimization process.  
  > **username_0**: @AyanDas Does it mean they update weights very slowly or do not converge?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-09. Score: 1
  
  The terms "*insufficient gradient*" or "*not strong enough gradient*" usually means that the magnitude of the gradient vector is too small or nearly zero that they can't drive the optimization properly.
  
  Not having sufficient gradient is similar to having a very low learning rate - they are not only slow (in terms of convergence) but also drifts the optimization in a poor direction and get stuck in local minima.
  
  
  ---
  
  