Answers.Body: <p>The two tech reports below both call RNNs explicitly &quot;recurrent net(work)s&quot;.</p>
  <ol>
  <li><p>Rumelhart, David E; Hinton, Geoffrey E, and Williams, Ronald J (Sept. 1985). <a href="https://apps.dtic.mil/dtic/tr/fulltext/u2/a164453.pdf" rel="nofollow noreferrer">Learning internal representations by error propagation</a>. Tech. rep. ICS 8504. San Diego, California: Institute for Cognitive Science, University of California.</p>
  </li>
  <li><p>Jordan, Michael I. (May 1986). <a href="https://www.osti.gov/biblio/6910294" rel="nofollow noreferrer">Serial order: a parallel distributed processing approach</a>. Tech. rep. ICS 8604. San Diego, California: Institute for Cognitive Science, University of California.</p>
  </li>
  </ol>
  <p><a href="https://grad.berkeley.edu/news/headlines/jordan-wins-rumelhart-prize/" rel="nofollow noreferrer">Jordan was a student of Rumelhart</a>, so I would lean on identifying <a href="https://apps.dtic.mil/dtic/tr/fulltext/u2/a164453.pdf" rel="nofollow noreferrer">1</a> as the paper introducing RNNs, with the caveat that the first sentence in the section &quot;Recurrent Nets&quot; of <a href="https://apps.dtic.mil/dtic/tr/fulltext/u2/a164453.pdf" rel="nofollow noreferrer">1</a> reads:</p>
  <blockquote>
  <p>We have thus far restricted ourselves to <em>feedforward</em> nets. This may
  seem like a substantial restriction, but as Minsky and Papert point
  out, there is, for every recurrent network, a feedforward network with
  identical behavior (over a finite period of time).</p>
  </blockquote>
  <p>This is interesting for two reasons:</p>
  <ol>
  <li>After this sentence, he then goes on to show how RNNs can be unrolled and the error propagated back. Not a full-fledged BPTT yet, though.</li>
  <li>The sentence shows that the idea of recurrence (and unrolling) has been around since at least 1969.</li>
  </ol>
  <p>Unfortunately, I don't have access to Minsky and Papert (1969), so I cannot follow this line any further.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-19T00:54:54.483Z
Answers.Text: You can find Minsky and Papert's book here: https://russell-davidson.arts.mcgill.ca/e706/Perceptrons.pdf. So, you may want to check that part. I have only found that this book mentions "Multilayer machines with loops" (section **13.2 Other Multilayer Machines**, pp. 231-232), so they were at least aware of the possibility of augmenting neural networks with recurrent connections, but I didn't find the term "recurrent" (although I didn't really search for it extensively).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-26T00:46:07.447Z
Answers.Text: As that link no longer seems to work, here a link to the Internet Archive version: https://archive.org/embed/perceptronsintro00mins
Answers.UserDisplayName: AlDante
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T11:29:23.710Z
Answers.LastActivityDate: 2021-01-18T23:30:20.397Z
Answers.LastEditDate: 2021-01-18T23:30:20.397Z
Answers.UserDisplayName: David Nemeskey
Answers.Body: <p>Warren McCulloch and Walter Pitts talk about recurrent neural nets in their paper McCulloch, W.S., Pitts, W. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5, 115–133 (1943). <a href="https://doi.org/10.1007/BF02478259" rel="nofollow noreferrer">https://doi.org/10.1007/BF02478259</a>.</p>
  <p>They finish their introduction with the paragraph:</p>
  <blockquote>
  <p>The nervous system contains many circular paths, whose activity so regenerates the excitation of any participant neuron that reference to time past becomes indefinite, although it still implies that afferent activity has realized one of a certain class of configurations over time. Precise specification of these implications by means of recursive functions, and determination of those that can be embodied in the activity of nervous nets, completes the theory.</p>
  </blockquote>
  <p>Their paper contains a section titled:</p>
  <blockquote>
  <ol start="2">
  <li>The Theory: Nets Without Circles.</li>
  </ol>
  </blockquote>
  <p>in which they introduce feed-forward (nets without cycles) and recurrent (nets with cycles) networks, and the next section, titled</p>
  <blockquote>
  <ol start="3">
  <li>The Theory: Nets with Circles.</li>
  </ol>
  </blockquote>
  <p>in which they prove a few theorems about recurrent neural networks.</p>
  <p>Marvin Minsky quotes them, and discusses recurrent neural networks extensively throughout his book, Computation: Finite and Infinite Machines (1967). Prentice Hall, ISBN: 0131655639,9780131655638</p>
  <p>I am not sure, are there earlier references.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-04T07:59:21.067Z
Answers.LastActivityDate: 2020-11-04T07:59:21.067Z
Answers.UserDisplayName: Vladislav Gladkikh
Answers.Body: <p>Hopfield networks, a special case of RNNs, were first proposed in 1982: <a href="https://www.pnas.org/content/79/8/2554" rel="noreferrer">https://www.pnas.org/content/79/8/2554</a></p>
  <p>Otherwise (shameless plug, I am the author) a non-technical timeline for NLP can be found here: <a href="https://blog.exxcellent.de/ki-machine-learning" rel="noreferrer">https://blog.exxcellent.de/ki-machine-learning</a></p>
  <p><a href="https://i.sstatic.net/aBMl0.jpg" rel="noreferrer"><img src="https://i.sstatic.net/aBMl0.jpg" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-04T11:49:22.767Z
Answers.Text: You say "a non-technical timeline for NLP can be found here:". Are you talking about a timeline for when RNNs were applied to NLP tasks?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-05T14:25:35.317Z
Answers.Text: No, the timeline shows the dates of the papers describing them. TR-808 was the introduction of the famous drum computer, referred to in the German text. 1982-86 were the papers on Hopfield networks and RNNs. 1995-97 the papers on LSTMs. And 1999 is the date the first GPU was launched. If you have corrections or comments, I would love to hear them.
Answers.UserDisplayName: AlDante
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-19T12:02:33.053Z
Answers.LastActivityDate: 2020-10-19T12:02:33.053Z
Answers.UserDisplayName: AlDante
Answers.Body: <p>According to <a href="https://arxiv.org/abs/1708.02709" rel="nofollow noreferrer">this meta paper</a>, &quot;vanilla&quot; RNN of today are based on Elman's work on networks with dynamic memory: <a href="https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1402_1" rel="nofollow noreferrer">Finding structure in time</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-17T14:24:17.247Z
Answers.LastActivityDate: 2021-08-17T14:24:17.247Z
Answers.UserDisplayName: Iván Mindlin
Body: <p>I was able to find <a href="https://www.bioinf.jku.at/publications/older/2604.pdf" rel="noreferrer">the original paper on LSTM</a>, but I was not able to find the paper that introduced &quot;vanilla&quot; RNNs. Where can I find it?</p>
  
Tags: <recurrent-neural-networks><long-short-term-memory><reference-request><papers>
Title: Where can I find the original paper that introduced RNNs?
UserDisplayName: Ahsan Tarique
ThreadText: # Where can I find the original paper that introduced RNNs?
  
  (Asked by: username_0 on 2018-09-30)
  
  I was able to find [the original paper on LSTM](https://www.bioinf.jku.at/publications/older/2604.pdf), but I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?
  
  **Tags:** recurrent-neural-networks, long-short-term-memory, reference-request, papers
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2020-03-10. Score: 13
  
  The two tech reports below both call RNNs explicitly "recurrent net(work)s".
  
  1. Rumelhart, David E; Hinton, Geoffrey E, and Williams, Ronald J (Sept. 1985). [Learning internal representations by error propagation](https://apps.dtic.mil/dtic/tr/fulltext/u2/a164453.pdf). Tech. rep. ICS 8504. San Diego, California: Institute for Cognitive Science, University of California.
  2. Jordan, Michael I. (May 1986). [Serial order: a parallel distributed processing approach](https://www.osti.gov/biblio/6910294). Tech. rep. ICS 8604. San Diego, California: Institute for Cognitive Science, University of California.
  
  [Jordan was a student of Rumelhart](https://grad.berkeley.edu/news/headlines/jordan-wins-rumelhart-prize/), so I would lean on identifying [1](https://apps.dtic.mil/dtic/tr/fulltext/u2/a164453.pdf) as the paper introducing RNNs, with the caveat that the first sentence in the section "Recurrent Nets" of [1](https://apps.dtic.mil/dtic/tr/fulltext/u2/a164453.pdf) reads:
  
  > We have thus far restricted ourselves to *feedforward* nets. This may
  > seem like a substantial restriction, but as Minsky and Papert point
  > out, there is, for every recurrent network, a feedforward network with
  > identical behavior (over a finite period of time).
  
  This is interesting for two reasons:
  
  1. After this sentence, he then goes on to show how RNNs can be unrolled and the error propagated back. Not a full-fledged BPTT yet, though.
  2. The sentence shows that the idea of recurrence (and unrolling) has been around since at least 1969.
  
  Unfortunately, I don't have access to Minsky and Papert (1969), so I cannot follow this line any further.
  
  > **Comments:**
  > **username_2**: You can find Minsky and Papert's book here: https://russell-davidson.arts.mcgill.ca/e706/Perceptrons.pdf. So, you may want to check that part. I have only found that this book mentions "Multilayer machines with loops" (section **13.2 Other Multilayer Machines**, pp. 231-232), so they were at least aware of the possibility of augmenting neural networks with recurrent connections, but I didn't find the term "recurrent" (although I didn't really search for it extensively).  
  > **username_3**: As that link no longer seems to work, here a link to the Internet Archive version: https://archive.org/embed/perceptronsintro00mins  
  
  ---
  
  ### Answer by username_3 on 2020-10-19. Score: 7
  
  Hopfield networks, a special case of RNNs, were first proposed in 1982: <https://www.pnas.org/content/79/8/2554>
  
  Otherwise (shameless plug, I am the author) a non-technical timeline for NLP can be found here: <https://blog.exxcellent.de/ki-machine-learning>
  
  [![enter image description here](https://i.sstatic.net/aBMl0.jpg)](https://i.sstatic.net/aBMl0.jpg)
  
  > **Comments:**
  > **username_2**: You say "a non-technical timeline for NLP can be found here:". Are you talking about a timeline for when RNNs were applied to NLP tasks?  
  > **username_3**: No, the timeline shows the dates of the papers describing them. TR-808 was the introduction of the famous drum computer, referred to in the German text. 1982-86 were the papers on Hopfield networks and RNNs. 1995-97 the papers on LSTMs. And 1999 is the date the first GPU was launched. If you have corrections or comments, I would love to hear them.  
  
  ---
  
  ### Answer by username_4 on 2020-11-04. Score: 4
  
  Warren McCulloch and Walter Pitts talk about recurrent neural nets in their paper McCulloch, W.S., Pitts, W. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5, 115–133 (1943). <https://doi.org/10.1007/BF02478259>.
  
  They finish their introduction with the paragraph:
  
  > The nervous system contains many circular paths, whose activity so regenerates the excitation of any participant neuron that reference to time past becomes indefinite, although it still implies that afferent activity has realized one of a certain class of configurations over time. Precise specification of these implications by means of recursive functions, and determination of those that can be embodied in the activity of nervous nets, completes the theory.
  
  Their paper contains a section titled:
  
  > 2. The Theory: Nets Without Circles.
  
  in which they introduce feed-forward (nets without cycles) and recurrent (nets with cycles) networks, and the next section, titled
  
  > 3. The Theory: Nets with Circles.
  
  in which they prove a few theorems about recurrent neural networks.
  
  Marvin Minsky quotes them, and discusses recurrent neural networks extensively throughout his book, Computation: Finite and Infinite Machines (1967). Prentice Hall, ISBN: 0131655639,9780131655638
  
  I am not sure, are there earlier references.
  
  
  ---
  
  ### Answer by username_5 on 2021-08-17. Score: 2
  
  According to [this meta paper](https://arxiv.org/abs/1708.02709), "vanilla" RNN of today are based on Elman's work on networks with dynamic memory: [Finding structure in time](https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1402_1)
  
  
  ---
  
  