Body: <p>It is recommended to apply gradient clipping by normalization in case of exploding gradients. The following quote is taken from <a href="https://ai.stackexchange.com/a/15607/18758">here</a> answer</p>
  <blockquote>
  <p>One way to assure it is exploding gradients is if the loss is unstable
  and not improving, or if loss shows NaN value during training.</p>
  <p>Apart from the <strong>usual gradient clipping</strong> and weights regularization that
  are recommended...</p>
  </blockquote>
  <p>But I want to know the effect of gradient clipping by normalization in the performance of the model in normal or general cases.</p>
  <p>Suppose I have a model and I run up to 800 epochs without gradient clipping because of the reason that there are no exploding gradients. If I run the same model with gradient clipping by norm, even if it is not necessary, then does the performance of the model decline?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-13T10:24:22.380Z
Comments.Text: [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://arxiv.org/abs/1905.11881) They use gradient clipping not just for RNNs, but also image recognition.
Comments.UserDisplayName: serali
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-13T11:46:58.497Z
Comments.Text: I think it's a good idea to tag your posts with more general tags, so that the context is immediately clear. For instance, in this case, gradient clipping is technique that is used for training neural networks with gradient descent, so, as I did, you could have added the tags that you see now. Of course, this is not a requirement and it's just my personal suggestion, but the tags can also clarify the context (as I said): if used only one tag and that related topic could occur in 2 different general contexts, that could make the question unclear.
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-learning><gradient-descent><gradient-clipping>
Title: What is the effect of gradient clipping by norm on the performance of a model?
UserDisplayName: hanugm
ThreadText: # What is the effect of gradient clipping by norm on the performance of a model?
  
  (Asked by: username_0 on 2021-10-13)
  
  It is recommended to apply gradient clipping by normalization in case of exploding gradients. The following quote is taken from [here](https://ai.stackexchange.com/a/15607/18758) answer
  
  > One way to assure it is exploding gradients is if the loss is unstable
  > and not improving, or if loss shows NaN value during training.
  >
  > Apart from the **usual gradient clipping** and weights regularization that
  > are recommended...
  
  But I want to know the effect of gradient clipping by normalization in the performance of the model in normal or general cases.
  
  Suppose I have a model and I run up to 800 epochs without gradient clipping because of the reason that there are no exploding gradients. If I run the same model with gradient clipping by norm, even if it is not necessary, then does the performance of the model decline?
  
  **Tags:** neural-networks, deep-learning, gradient-descent, gradient-clipping
  > **Comments:**
  > **username_1**: [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://arxiv.org/abs/1905.11881) They use gradient clipping not just for RNNs, but also image recognition.  
  > **username_2**: I think it's a good idea to tag your posts with more general tags, so that the context is immediately clear. For instance, in this case, gradient clipping is technique that is used for training neural networks with gradient descent, so, as I did, you could have added the tags that you see now. Of course, this is not a requirement and it's just my personal suggestion, but the tags can also clarify the context (as I said): if used only one tag and that related topic could occur in 2 different general contexts, that could make the question unclear.  
  
  ---
  
  ## 0 answers
  
  