Answers.Body: <p>A <em>recurrent neural network</em> (RNN) is an <a href="https://ai.stackexchange.com/a/11978/2444">artificial neural network</a> that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks.</p>
  
  <p>An RNN can be trained using <em>back-propagation through time</em> (BBTT), such that these backward or self-connections "memorise" previously seen inputs. Hence, these connections are mainly used to track temporal relations between elements of a sequence of inputs, which makes RNNs well suited to sequence prediction and similar tasks.</p>
  
  <p>There are several RNN models: for example, RNNs with LSTM or GRU units. LSTM (or GRU) is an RNN whose single units perform a more complex transformation than a unit in a "plain RNN", which performs a linear transformation of the input followed by the application of a non-linear function (e.g. ReLU) to this linear transformation. In theory, "plain RNN" are as powerful as RNNs with LSTM units. In practice, they suffer from the "vanishing and exploding gradients" problem. Hence, in practice, LSTMs (or similar sophisticated recurrent units) are used.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-28T17:39:11.357Z
Answers.LastActivityDate: 2019-04-28T17:39:11.357Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Recurrent Neural Network(RNN):</p>
  <ul>
  <li>can remember the output of the previous step to use it in the next step.</li>
  <li>is used for NLP(Natural Language Processing). *NLP is the technology which enables computers to understand and communicate with human language.</li>
  <li>is <a href="https://pytorch.org/docs/stable/generated/torch.nn.RNN.html" rel="nofollow noreferrer">RNN()</a> in PyTorch.</li>
  </ul>
  <p>In addition, there is LSTM(Long Short-Term Memory) which is the improved version of RNN.</p>
  <p>LSTM(Long Short-Term Memory):</p>
  <ul>
  <li>can remember the output of the previous steps to use them in the next step while RNN can remember the output of only the previous step.</li>
  <li>has more ability than Recurrent Layer(RNN).</li>
  <li>is slower to train data than RNN because the computation is more complex.</li>
  <li>is used for NLP.</li>
  <li>is <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM" rel="nofollow noreferrer">LSTM()</a> in PyTorch.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-13T01:41:51.357Z
Answers.LastActivityDate: 2024-06-13T01:41:51.357Z
Answers.UserDisplayName: Super Kai - Kazuya Ito
Answers.Body: <p>Recurrent neural networks (RNNs) are a class of artificial neural network
  architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store information. </p>
  
  <p>Difference with traditional Neural networks using pictures from <a href="https://www.cs.toronto.edu/~graves/preprint.pdf" rel="noreferrer">this book</a>:</p>
  
  <p><a href="https://i.sstatic.net/ULBqq.png" rel="noreferrer"><img src="https://i.sstatic.net/ULBqq.png" alt="enter image description here"></a></p>
  
  <p>And, an RNN:</p>
  
  <p><a href="https://i.sstatic.net/qobNg.png" rel="noreferrer"><img src="https://i.sstatic.net/qobNg.png" alt="enter image description here"></a></p>
  
  <p>Notice the difference -- feedforward neural networks' connections
  do not form cycles. If we relax this condition, and allow cyclical
  connections as well, we obtain recurrent neural networks (RNNs). You can see that in the hidden layer of the architecture.</p>
  
  <p>While the difference between a multilayer perceptron and an RNN may seem
  trivial, the implications for sequence learning are far-reaching. An MLP can only
  map from <strong>input to output vectors</strong>, whereas an RNN can in principle map from
  the <strong>entire history of previous inputs to each output</strong>. Indeed, the equivalent
  result to the universal approximation theory for MLPs is that an RNN with a
  sufficient number of hidden units can approximate any measurable sequence-to-sequence
  mapping to arbitrary accuracy.</p>
  
  <p>Important takeaway:</p>
  
  <p><strong>The recurrent connections allow a 'memory' of previous inputs to persist in the
  network's internal state, and thereby influence the network output.</strong></p>
  
  <p>Talking in terms of advantages is not appropriate as they both are state-of-the-art and are particularly good at certain tasks. A broad category of tasks that RNN excel at is:</p>
  
  <h2>Sequence Labelling</h2>
  
  <p>The goal of sequence labelling is to assign sequences of labels, drawn from a fixed alphabet, to sequences of input data.</p>
  
  <p>Ex: Transcribe a sequence of acoustic features with spoken words (speech recognition), or a sequence of video frames with hand gestures (gesture recognition).</p>
  
  <p>Some of the sub-tasks in sequence labelling are:</p>
  
  <p><strong>Sequence Classification</strong></p>
  
  <p>Label sequences are constrained to be of length one. This is referred to as sequence classification, since each input sequence is assigned to a single class. Examples of sequence classification task include the identification of a single spoken work and the recognition of an individual
  handwritten letter.</p>
  
  <p><strong>Segment Classification</strong></p>
  
  <p>Segment classification refers to those tasks where the target sequences consist
  of multiple labels, but the locations of the labels -- that is, the positions of the input segments to which the labels apply -- are known in advance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-29T07:12:07.653Z
Answers.LastActivityDate: 2019-04-29T13:54:41.947Z
Answers.LastEditDate: 2019-04-29T13:54:41.947Z
Answers.UserDisplayName: naive
Body: <p>Surprisingly, this wasn't asked before - at least I didn't find anything besides some <a href="https://ai.stackexchange.com/q/11717/2444">vaguely related questions.</a></p>
  
  <p>So, what is a recurrent neural network, and what are their advantages over regular (or feed-forward) neural networks?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-24T17:46:12.603Z
Comments.Text: See the blog post [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Christopher Olah.
Comments.UserDisplayName: MrE
Tags: <neural-networks><comparison><recurrent-neural-networks><definitions><feedforward-neural-networks>
Title: What is a recurrent neural network?
UserDisplayName: olinarr
ThreadText: # What is a recurrent neural network?
  
  (Asked by: username_0 on 2019-04-28)
  
  Surprisingly, this wasn't asked before - at least I didn't find anything besides some [vaguely related questions.](https://ai.stackexchange.com/q/11717/2444)
  
  So, what is a recurrent neural network, and what are their advantages over regular (or feed-forward) neural networks?
  
  **Tags:** neural-networks, comparison, recurrent-neural-networks, definitions, feedforward-neural-networks
  > **Comments:**
  > **username_1**: See the blog post [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Christopher Olah.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2019-04-29. Score: 9 | Accepted answer
  
  Recurrent neural networks (RNNs) are a class of artificial neural network
  architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store information.
  
  Difference with traditional Neural networks using pictures from [this book](https://www.cs.toronto.edu/~graves/preprint.pdf):
  
  [![enter image description here](https://i.sstatic.net/ULBqq.png)](https://i.sstatic.net/ULBqq.png)
  
  And, an RNN:
  
  [![enter image description here](https://i.sstatic.net/qobNg.png)](https://i.sstatic.net/qobNg.png)
  
  Notice the difference -- feedforward neural networks' connections
  do not form cycles. If we relax this condition, and allow cyclical
  connections as well, we obtain recurrent neural networks (RNNs). You can see that in the hidden layer of the architecture.
  
  While the difference between a multilayer perceptron and an RNN may seem
  trivial, the implications for sequence learning are far-reaching. An MLP can only
  map from **input to output vectors**, whereas an RNN can in principle map from
  the **entire history of previous inputs to each output**. Indeed, the equivalent
  result to the universal approximation theory for MLPs is that an RNN with a
  sufficient number of hidden units can approximate any measurable sequence-to-sequence
  mapping to arbitrary accuracy.
  
  Important takeaway:
  
  **The recurrent connections allow a 'memory' of previous inputs to persist in the
  network's internal state, and thereby influence the network output.**
  
  Talking in terms of advantages is not appropriate as they both are state-of-the-art and are particularly good at certain tasks. A broad category of tasks that RNN excel at is:
  
  ## Sequence Labelling
  
  The goal of sequence labelling is to assign sequences of labels, drawn from a fixed alphabet, to sequences of input data.
  
  Ex: Transcribe a sequence of acoustic features with spoken words (speech recognition), or a sequence of video frames with hand gestures (gesture recognition).
  
  Some of the sub-tasks in sequence labelling are:
  
  **Sequence Classification**
  
  Label sequences are constrained to be of length one. This is referred to as sequence classification, since each input sequence is assigned to a single class. Examples of sequence classification task include the identification of a single spoken work and the recognition of an individual
  handwritten letter.
  
  **Segment Classification**
  
  Segment classification refers to those tasks where the target sequences consist
  of multiple labels, but the locations of the labels -- that is, the positions of the input segments to which the labels apply -- are known in advance.
  
  
  ---
  
  ### Answer by username_2 on 2019-04-28. Score: 11
  
  A *recurrent neural network* (RNN) is an [artificial neural network](https://ai.stackexchange.com/a/11978/2444) that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks.
  
  An RNN can be trained using *back-propagation through time* (BBTT), such that these backward or self-connections "memorise" previously seen inputs. Hence, these connections are mainly used to track temporal relations between elements of a sequence of inputs, which makes RNNs well suited to sequence prediction and similar tasks.
  
  There are several RNN models: for example, RNNs with LSTM or GRU units. LSTM (or GRU) is an RNN whose single units perform a more complex transformation than a unit in a "plain RNN", which performs a linear transformation of the input followed by the application of a non-linear function (e.g. ReLU) to this linear transformation. In theory, "plain RNN" are as powerful as RNNs with LSTM units. In practice, they suffer from the "vanishing and exploding gradients" problem. Hence, in practice, LSTMs (or similar sophisticated recurrent units) are used.
  
  
  ---
  
  ### Answer by username_3 on 2024-06-13. Score: 0
  
  Recurrent Neural Network(RNN):
  
  * can remember the output of the previous step to use it in the next step.
  * is used for NLP(Natural Language Processing). \*NLP is the technology which enables computers to understand and communicate with human language.
  * is [RNN()](https://pytorch.org/docs/stable/generated/torch.nn.RNN.html) in PyTorch.
  
  In addition, there is LSTM(Long Short-Term Memory) which is the improved version of RNN.
  
  LSTM(Long Short-Term Memory):
  
  * can remember the output of the previous steps to use them in the next step while RNN can remember the output of only the previous step.
  * has more ability than Recurrent Layer(RNN).
  * is slower to train data than RNN because the computation is more complex.
  * is used for NLP.
  * is [LSTM()](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM) in PyTorch.
  
  
  ---
  
  