<p>Gradient clipping is a technique that tackles exploding gradients. The idea of gradient clipping is very simple: If the gradient gets too large, we rescale it to keep it small. More precisely,</p>
<p><span class="math-container">$$
\text{if } \Vert \mathbf{g} \Vert \geq c, \text{then }
\mathbf{g} \leftarrow c \frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}
$$</span></p>
<p>where <span class="math-container">$c$</span> is a hyperparameter, <span class="math-container">$\mathbf{g}$</span> is the gradient, and <span class="math-container">$\Vert \mathbf{g} \Vert$</span> is the norm of <span class="math-container">$\mathbf{g}$</span>.</p>
<p>Since <span class="math-container">$\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}$</span> is a unit vector, after rescaling the new <span class="math-container">$\mathbf{g}$</span> will have norm <span class="math-container">$c$</span>.</p>
<p>Note that if <span class="math-container">$\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert} &lt; c$</span> , then we don’t need to do anything.</p>
<p>Check <a href="https://towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48" rel="nofollow noreferrer">this article </a> for more information</p>

CC BY-SA 4.0
2021-12-27T02:16:32.327Z
This appears to be copy-pasted from [another source](https://towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48), without appropriate attribution.  We have certain rules that we require you to follow whenever copying from another source: https://ai.stackexchange.com/help/referencing.  Please [edit] your answer to follow them.
D.W.
CC BY-SA 4.0
2021-10-10T04:58:26.003Z
2021-11-01T11:14:56.897Z
2021-11-01T11:14:56.897Z
Archana David
<p>Consider the <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html" rel="nofollow noreferrer">following description</a> regarding gradient clipping in PyTorch</p>
<pre><code>torch.nn.utils.clip_grad_norm_(parameters, max_norm, norm_type=2.0, error_if_nonfinite=False)
</code></pre>
<blockquote>
<p><strong>Clips</strong> gradient norm of an iterable of parameters.</p>
<p>The norm is computed over all gradients together as if they were
concatenated into a single vector. Gradients are modified in-place.</p>
</blockquote>
<p>Let the weights and gradients, for loss function <span class="math-container">$L$</span>, of the model, be given as below</p>
<p><span class="math-container">\begin{align}
w 
&amp;= [w_1, w_2, w_3, \cdots, w_n] \\
\triangledown 
&amp;= [\triangledown_1, \triangledown_2, \triangledown_3, \cdots, \triangledown_n] \text{, where } \triangledown_i = \dfrac{\partial L}{\partial w_i} \text{ and } 1 \le i \le n
\end{align}</span></p>
<p>From the description, we need to compute gradient norm, i.e. <span class="math-container">$||\triangledown||$</span>.</p>
<p>How to proceed after the step of finding the gradient norm? What is meant by clipping the gradient norm mathematically?</p>

<deep-learning><pytorch><gradient-clipping>
What exactly happens in gradient clipping by norm?
hanugm
# What exactly happens in gradient clipping by norm?

(Asked by: username_0 on 2021-10-10)

Consider the following description regarding gradient clipping in PyTorch
torch.nn.utils.clip_grad_norm_(parameters, max_norm, norm_type=2.0, error_if_nonfinite=False)


Clips gradient norm of an iterable of parameters.
The norm is computed over all gradients together as if they were
concatenated into a single vector. Gradients are modified in-place.

Let the weights and gradients, for loss function $L$, of the model, be given as below
\begin{align}
w 
&= [w_1, w_2, w_3, \cdots, w_n] \\
\triangledown 
&= [\triangledown_1, \triangledown_2, \triangledown_3, \cdots, \triangledown_n] \text{, where } \triangledown_i = \dfrac{\partial L}{\partial w_i} \text{ and } 1 \le i \le n
\end{align}
From the description, we need to compute gradient norm, i.e. $||\triangledown||$.
How to proceed after the step of finding the gradient norm? What is meant by clipping the gradient norm mathematically?

**Tags:** deep-learning, pytorch, gradient-clipping


---

## 1 answers

### Answer by username_1 on 2021-10-10. Score: 4

Gradient clipping is a technique that tackles exploding gradients. The idea of gradient clipping is very simple: If the gradient gets too large, we rescale it to keep it small. More precisely,

$$
\text{if } \Vert \mathbf{g} \Vert \geq c, \text{then }
\mathbf{g} \leftarrow c \frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}
$$

where $c$ is a hyperparameter, $\mathbf{g}$ is the gradient, and $\Vert \mathbf{g} \Vert$ is the norm of $\mathbf{g}$.

Since $\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}$ is a unit vector, after rescaling the new $\mathbf{g}$ will have norm $c$.

Note that if $\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert} < c$ , then we don’t need to do anything.

Check [this article](https://towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48)  for more information

> **Comments:**
> **username_2**: This appears to be copy-pasted from [another source](https://towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48), without appropriate attribution.  We have certain rules that we require you to follow whenever copying from another source: https://ai.stackexchange.com/help/referencing.  Please [edit] your answer to follow them.  

---

