Answers.Body: <h2>Short Answer</h2>
  
  <p>Tomas Mikolov's mention of gradient clipping in a  single paragraph of his PhD thesis in 2012 is the first appearance in the literature.</p>
  
  <h2>Long Answer</h2>
  
  <p>The first source (Mikolov, 2012) in the <em>Deep Learning</em> book is Mikolov's PhD thesis and can be found <a href="https://www.fit.vutbr.cz/~imikolov/rnnlm/thesis.pdf" rel="nofollow noreferrer">here</a>. The end of section 3.2.2 is where gradient clipping is discussed, only it's called <em>turncating</em>. </p>
  
  <blockquote>
    <p>... The exploding gradient problem has
    been described in [4].</p>
    
    <p>A simple solution to the exploding gradient problem is to truncate values of the gradients. In my experiments, I did limit maximum size of gradients of errors that get accumulated in the hidden neurons to be in a range &lt; −15; 15 >. This greatly increases stability
    of the training, and otherwise it would not be possible to train RNN LMs successfully on
    large data sets.<br>
    ...<br>
    [4] Y. Bengio, P. Simard, P. Frasconi. Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks, 5, 157-166, 1994. </p>
  </blockquote>
  
  <p>A search of the referenced paper <a href="http://ai.dinfo.unifi.it/paolo/ps/tnn-94-gradient.pdf" rel="nofollow noreferrer">[4]</a> shows that it does describe the problem as Mikolov said, but it does not present gradient clipping as a solution.   </p>
  
  <p>So I had a look at the second source <em>Deep Learning</em> mentioned: <a href="https://arxiv.org/abs/1211.5063" rel="nofollow noreferrer"><em>On the difficulty of training Recurrent Neural Networks</em></a>. It directly cites Mikolov as having proposed clipping: </p>
  
  <blockquote>
    <p>We would make a final note about the approach proposed by Tomas Mikolov in his PhD thesis (Mikolov, 2012) (and implicitly used in the state of the art results on language modelling (Mikolov et al., 2011)). It involves clipping the gradient’s temporal components element-wise (clipping an entry when it exceeds in absolute value a fixed threshold). Clipping has been shown to do well in practice and it forms the backbone of our approach.</p>
  </blockquote>
  
  <p>I thought about emailing Mikolov to verify that his thesis was the origin of the idea. But then I noticed that he is a co-author of this paper which cites him as proposing it! Though I still wonder if it was commonly used in practice before 
  even though it had not been published. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-05T17:40:10.933Z
Answers.Text: Maybe have also a look at the LSTM paper. It probably talks about the topic too.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-05T19:25:47.303Z
Answers.Text: @nbro If you're talking about the original 1997 paper by Hochreiter and Schmidhuber, there is no mention of 'clip'/'clipping' and all the instances of 'truncate' appear to be in reference to truncating the BPTT algorithm, not the gradients themselves.
Answers.UserDisplayName: Jeremiah England
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-05T17:30:33.197Z
Answers.LastActivityDate: 2020-04-05T17:30:33.197Z
Answers.UserDisplayName: Jeremiah England
Body: <p>The <a href="https://www.deeplearningbook.org/contents/rnn.html" rel="nofollow noreferrer"><em>Deep Learning</em> book</a> mentions that it's been used for years but the oldest sources it mentions are from 2012: </p>
  
  <blockquote>
    <p>A simple type of solution has been in use by practitioners for many years: clipping the gradient. There are diﬀerent instances of this idea (Mikolov, 2012; Pascanu et al., 2013). One option is to clip the parameter gradient from a mini-batch element-wise (Mikolov, 2012), just before the parameter update. Another is to clip the <span class="math-container">$||g||$</span> of the gradient <span class="math-container">$g$</span> (Pascanu et al., 2013) just before the parameter update  </p>
  </blockquote>
  
  <p>But I find it hard to believe that the first uses and mentions of gradient clipping are from 2012. Does anyone know the origins of the solution? </p>
  
Tags: <reference-request><papers><history><exploding-gradient-problem>
Title: Which work originally introduced gradient clipping?
UserDisplayName: Jeremiah England
ThreadText: # Which work originally introduced gradient clipping?
  
  (Asked by: username_0 on 2020-03-31)
  
  The [*Deep Learning* book](https://www.deeplearningbook.org/contents/rnn.html) mentions that it's been used for years but the oldest sources it mentions are from 2012:
  
  > A simple type of solution has been in use by practitioners for many years: clipping the gradient. There are diﬀerent instances of this idea (Mikolov, 2012; Pascanu et al., 2013). One option is to clip the parameter gradient from a mini-batch element-wise (Mikolov, 2012), just before the parameter update. Another is to clip the $||g||$ of the gradient $g$ (Pascanu et al., 2013) just before the parameter update
  
  But I find it hard to believe that the first uses and mentions of gradient clipping are from 2012. Does anyone know the origins of the solution?
  
  **Tags:** reference-request, papers, history, exploding-gradient-problem
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-04-05. Score: 2 | Accepted answer
  
  ## Short Answer
  
  Tomas Mikolov's mention of gradient clipping in a single paragraph of his PhD thesis in 2012 is the first appearance in the literature.
  
  ## Long Answer
  
  The first source (Mikolov, 2012) in the *Deep Learning* book is Mikolov's PhD thesis and can be found [here](https://www.fit.vutbr.cz/~imikolov/rnnlm/thesis.pdf). The end of section 3.2.2 is where gradient clipping is discussed, only it's called *turncating*.
  
  > ... The exploding gradient problem has
  > been described in [4].
  >
  > A simple solution to the exploding gradient problem is to truncate values of the gradients. In my experiments, I did limit maximum size of gradients of errors that get accumulated in the hidden neurons to be in a range < −15; 15 >. This greatly increases stability
  > of the training, and otherwise it would not be possible to train RNN LMs successfully on
  > large data sets.  
  > ...  
  > [4] Y. Bengio, P. Simard, P. Frasconi. Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks, 5, 157-166, 1994.
  
  A search of the referenced paper [[4]](http://ai.dinfo.unifi.it/paolo/ps/tnn-94-gradient.pdf) shows that it does describe the problem as Mikolov said, but it does not present gradient clipping as a solution.
  
  So I had a look at the second source *Deep Learning* mentioned: [*On the difficulty of training Recurrent Neural Networks*](https://arxiv.org/abs/1211.5063). It directly cites Mikolov as having proposed clipping:
  
  > We would make a final note about the approach proposed by Tomas Mikolov in his PhD thesis (Mikolov, 2012) (and implicitly used in the state of the art results on language modelling (Mikolov et al., 2011)). It involves clipping the gradient’s temporal components element-wise (clipping an entry when it exceeds in absolute value a fixed threshold). Clipping has been shown to do well in practice and it forms the backbone of our approach.
  
  I thought about emailing Mikolov to verify that his thesis was the origin of the idea. But then I noticed that he is a co-author of this paper which cites him as proposing it! Though I still wonder if it was commonly used in practice before
  even though it had not been published.
  
  > **Comments:**
  > **username_1**: Maybe have also a look at the LSTM paper. It probably talks about the topic too.  
  > **username_0**: @username_1 If you're talking about the original 1997 paper by Hochreiter and Schmidhuber, there is no mention of 'clip'/'clipping' and all the instances of 'truncate' appear to be in reference to truncating the BPTT algorithm, not the gradients themselves.  
  
  ---
  
  