Answers.Body: <p>In an RNN, the output of the previous state is passed as an input to the current state. Intuitively, there is a temporal (time-based) relationship in the way in which input is processed in an RNN. It can understand how the current state was achieved on the basis of the previous values, i.e value at time-step <span class="math-container">$t$</span> is a result of value at time-steps <span class="math-container">$t-1, t-2$</span>, and so on.</p>
  <p>In a DNN, there is no temporal relationship in the way input is processed. Values at time-steps <span class="math-container">$t, t-1, t-2, \dots $</span> are all treated distinctly and not as a continuation of the previous time-step values.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-29T00:45:19.830Z
Answers.Text: Doesn't the MLP account for this relationship in the hidden layers? I mean the matrix multiplication to go from the input to the first hidden layer includes all the time steps.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-20T09:53:01.590Z
Answers.LastActivityDate: 2020-10-02T19:22:16.633Z
Answers.LastEditDate: 2020-10-02T19:22:16.633Z
Answers.UserDisplayName: Dishant Sheth
Answers.Body: <p>As noted in <a href="https://www.nature.com/articles/nature14539.epdf" rel="nofollow noreferrer">LeCun et al.'s Deep Learning paper</a>:</p>
  <blockquote>
  <p>RNNs, once unfolded in time ... can be seen as very deep feedforward networks in which all the layers share the same weights</p>
  </blockquote>
  <p>So, if we ignore how easy they are to train, there is theoretically no real advantage of RNNs over MLPs, on any task, including time series modeling.</p>
  <p>Perhaps the key advantage of RNNs is that they share parameters over time.  That means they have fewer parameters, and the parameters they do have get used more often.  This makes them easier to train (and also allows handling variable-length input sequences, which, as you noted, does not apply when forecasting from a fixed historical window).</p>
  <p>Also, the &quot;compression&quot; of the historical information into a state that can pass forward in time, as noted in <a href="https://www.quora.com/Whats-the-difference-between-a-single-output-RNN-and-an-MLP-whose-input-data-contains-all-of-the-features-of-the-given-time-steps" rel="nofollow noreferrer">this excellent answer on quora</a>, may force RNNs to learn general patterns that may generalize better to unseen data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-07T20:56:00.510Z
Answers.LastActivityDate: 2022-10-07T20:56:00.510Z
Answers.UserDisplayName: shaneb
Answers.Body: <p>RNNs have the ability to hold a state, that means the model can learn which information it wants to save and what to delete based on ordering and how you designed the creation and passing of the state (probably worth looking into what an LSTM is), while this would be alot more difficult for a sliding MLP to do (you can think of a sliding MLP as a stateless RNN). Also a sliding MLP would be alot more computation since it needs to recompute an entire context window for each new output, while an RNN could just use the previous state and only do processing on a new singular input or a smaller context window.</p>
  
  <p>Hope that helps!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-16T19:47:05.307Z
Answers.LastActivityDate: 2019-05-16T19:47:05.307Z
Answers.UserDisplayName: mshlis
Body: <p>Understandably RNNs are very good at solving problems involving audio, video and text processing due to the arbitrary input's length of this sort of data.</p>
  <p>What I don't understand is why RNNs are also superior at predicting time series data and why we use them over simple MLP DNNs.</p>
  <p>Say I wanted to predict what the value in the time series is at <span class="math-container">$t+1$</span>. I would take a window of, let's say, <span class="math-container">$t-50, t-49, \dots, t$</span>, and then feed loads of sampled training data into a network. I could either choose to have a single LSTM unit remembering the entire window and basing the predictions on that, or I could simply make a 50 neuron wide MLP network.</p>
  <p>What exactly is it about RNNs that makes them better in this scenario or any scenario for that matter?</p>
  <p>I understand that the LSTM would have substantially fewer weights (in this scenario) and should be less computationally intensive, but, apart from that, I don't see any difference in these two methods.</p>
  
Tags: <neural-networks><recurrent-neural-networks><long-short-term-memory><time-series><feedforward-neural-networks>
Title: Why are RNNs better than MLPs at predicting time series data?
UserDisplayName: Dema Ushchapovskyy
ThreadText: # Why are RNNs better than MLPs at predicting time series data?
  
  (Asked by: username_0 on 2019-05-16)
  
  Understandably RNNs are very good at solving problems involving audio, video and text processing due to the arbitrary input's length of this sort of data.
  
  What I don't understand is why RNNs are also superior at predicting time series data and why we use them over simple MLP DNNs.
  
  Say I wanted to predict what the value in the time series is at $t+1$. I would take a window of, let's say, $t-50, t-49, \dots, t$, and then feed loads of sampled training data into a network. I could either choose to have a single LSTM unit remembering the entire window and basing the predictions on that, or I could simply make a 50 neuron wide MLP network.
  
  What exactly is it about RNNs that makes them better in this scenario or any scenario for that matter?
  
  I understand that the LSTM would have substantially fewer weights (in this scenario) and should be less computationally intensive, but, apart from that, I don't see any difference in these two methods.
  
  **Tags:** neural-networks, recurrent-neural-networks, long-short-term-memory, time-series, feedforward-neural-networks
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-05-20. Score: 1 | Accepted answer
  
  In an RNN, the output of the previous state is passed as an input to the current state. Intuitively, there is a temporal (time-based) relationship in the way in which input is processed in an RNN. It can understand how the current state was achieved on the basis of the previous values, i.e value at time-step $t$ is a result of value at time-steps $t-1, t-2$, and so on.
  
  In a DNN, there is no temporal relationship in the way input is processed. Values at time-steps $t, t-1, t-2, \dots $ are all treated distinctly and not as a continuation of the previous time-step values.
  
  > **Comments:**
  > **username_2**: Doesn't the MLP account for this relationship in the hidden layers? I mean the matrix multiplication to go from the input to the first hidden layer includes all the time steps.  
  
  ---
  
  ### Answer by username_3 on 2022-10-07. Score: 1
  
  As noted in [LeCun et al.'s Deep Learning paper](https://www.nature.com/articles/nature14539.epdf):
  
  > RNNs, once unfolded in time ... can be seen as very deep feedforward networks in which all the layers share the same weights
  
  So, if we ignore how easy they are to train, there is theoretically no real advantage of RNNs over MLPs, on any task, including time series modeling.
  
  Perhaps the key advantage of RNNs is that they share parameters over time. That means they have fewer parameters, and the parameters they do have get used more often. This makes them easier to train (and also allows handling variable-length input sequences, which, as you noted, does not apply when forecasting from a fixed historical window).
  
  Also, the "compression" of the historical information into a state that can pass forward in time, as noted in [this excellent answer on quora](https://www.quora.com/Whats-the-difference-between-a-single-output-RNN-and-an-MLP-whose-input-data-contains-all-of-the-features-of-the-given-time-steps), may force RNNs to learn general patterns that may generalize better to unseen data.
  
  
  ---
  
  ### Answer by username_4 on 2019-05-16. Score: 0
  
  RNNs have the ability to hold a state, that means the model can learn which information it wants to save and what to delete based on ordering and how you designed the creation and passing of the state (probably worth looking into what an LSTM is), while this would be alot more difficult for a sliding MLP to do (you can think of a sliding MLP as a stateless RNN). Also a sliding MLP would be alot more computation since it needs to recompute an entire context window for each new output, while an RNN could just use the previous state and only do processing on a new singular input or a smaller context window.
  
  Hope that helps!
  
  
  ---
  
  