Answers.Body: <p>Gradient clipping is a technique that tackles exploding gradients. The idea of gradient clipping is very simple: If the gradient gets too large, we rescale it to keep it small. More precisely,</p>
  <p><span class="math-container">$$
  \text{if } \Vert \mathbf{g} \Vert \geq c, \text{then }
  \mathbf{g} \leftarrow c \frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}
  $$</span></p>
  <p>where <span class="math-container">$c$</span> is a hyperparameter, <span class="math-container">$\mathbf{g}$</span> is the gradient, and <span class="math-container">$\Vert \mathbf{g} \Vert$</span> is the norm of <span class="math-container">$\mathbf{g}$</span>.</p>
  <p>Since <span class="math-container">$\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}$</span> is a unit vector, after rescaling the new <span class="math-container">$\mathbf{g}$</span> will have norm <span class="math-container">$c$</span>.</p>
  <p>Note that if <span class="math-container">$\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert} &lt; c$</span> , then we don’t need to do anything.</p>
  <p>Check <a href="https://towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48" rel="nofollow noreferrer">this article </a> for more information</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-27T02:16:32.327Z
Answers.Text: This appears to be copy-pasted from [another source](https://towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48), without appropriate attribution.  We have certain rules that we require you to follow whenever copying from another source: https://ai.stackexchange.com/help/referencing.  Please [edit] your answer to follow them.
Answers.UserDisplayName: D.W.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-10T04:58:26.003Z
Answers.LastActivityDate: 2021-11-01T11:14:56.897Z
Answers.LastEditDate: 2021-11-01T11:14:56.897Z
Answers.UserDisplayName: Archana David
Body: <p>Consider the <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html" rel="nofollow noreferrer">following description</a> regarding gradient clipping in PyTorch</p>
  <pre><code>torch.nn.utils.clip_grad_norm_(parameters, max_norm, norm_type=2.0, error_if_nonfinite=False)
  </code></pre>
  <blockquote>
  <p><strong>Clips</strong> gradient norm of an iterable of parameters.</p>
  <p>The norm is computed over all gradients together as if they were
  concatenated into a single vector. Gradients are modified in-place.</p>
  </blockquote>
  <p>Let the weights and gradients, for loss function <span class="math-container">$L$</span>, of the model, be given as below</p>
  <p><span class="math-container">\begin{align}
  w 
  &amp;= [w_1, w_2, w_3, \cdots, w_n] \\
  \triangledown 
  &amp;= [\triangledown_1, \triangledown_2, \triangledown_3, \cdots, \triangledown_n] \text{, where } \triangledown_i = \dfrac{\partial L}{\partial w_i} \text{ and } 1 \le i \le n
  \end{align}</span></p>
  <p>From the description, we need to compute gradient norm, i.e. <span class="math-container">$||\triangledown||$</span>.</p>
  <p>How to proceed after the step of finding the gradient norm? What is meant by clipping the gradient norm mathematically?</p>
  
Tags: <deep-learning><pytorch><gradient-clipping>
Title: What exactly happens in gradient clipping by norm?
UserDisplayName: hanugm
ThreadText: # What exactly happens in gradient clipping by norm?
  
  (Asked by: username_0 on 2021-10-10)
  
  Consider the [following description](https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html) regarding gradient clipping in PyTorch
  
  ```
  torch.nn.utils.clip_grad_norm_(parameters, max_norm, norm_type=2.0, error_if_nonfinite=False)
  
  ```
  
  > **Clips** gradient norm of an iterable of parameters.
  >
  > The norm is computed over all gradients together as if they were
  > concatenated into a single vector. Gradients are modified in-place.
  
  Let the weights and gradients, for loss function $L$, of the model, be given as below
  
  \begin{align}
  w
  &= [w\_1, w\_2, w\_3, \cdots, w\_n] \\
  \triangledown
  &= [\triangledown\_1, \triangledown\_2, \triangledown\_3, \cdots, \triangledown\_n] \text{, where } \triangledown\_i = \dfrac{\partial L}{\partial w\_i} \text{ and } 1 \le i \le n
  \end{align}
  
  From the description, we need to compute gradient norm, i.e. $||\triangledown||$.
  
  How to proceed after the step of finding the gradient norm? What is meant by clipping the gradient norm mathematically?
  
  **Tags:** deep-learning, pytorch, gradient-clipping
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-10-10. Score: 4
  
  Gradient clipping is a technique that tackles exploding gradients. The idea of gradient clipping is very simple: If the gradient gets too large, we rescale it to keep it small. More precisely,
  
  $$
  \text{if } \Vert \mathbf{g} \Vert \geq c, \text{then }
  \mathbf{g} \leftarrow c \frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}
  $$
  
  where $c$ is a hyperparameter, $\mathbf{g}$ is the gradient, and $\Vert \mathbf{g} \Vert$ is the norm of $\mathbf{g}$.
  
  Since $\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}$ is a unit vector, after rescaling the new $\mathbf{g}$ will have norm $c$.
  
  Note that if $\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert} < c$ , then we don’t need to do anything.
  
  Check [this article](https://towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48)  for more information
  
  > **Comments:**
  > **username_2**: This appears to be copy-pasted from [another source](https://towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48), without appropriate attribution.  We have certain rules that we require you to follow whenever copying from another source: https://ai.stackexchange.com/help/referencing.  Please [edit] your answer to follow them.  
  
  ---
  
  