Answers.Body: <p>You don't necessarily need to roll out the inputs to an RNN; doing so makes it easier to optimize computation (if the sequence length is the same each batch), but it's not a necessity. Furthermore, RNNs (and, incidentally, the brain) doesn't necessarily remember the input history as is; rather, the history is encoded via the RNN's cell state (or states, in the case of LSTMs and other RNN cell architectures with multiple states). Neural Turing Machines (NMTs) and Differential neural computers expand on that concept by also using a larger "memory" storage (in the form of a matrix).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-31T19:28:32.923Z
Answers.Text: I don't think that's true. To train an RNN you definitely need to roll it out. To use a pre-trained RNN you don't.
Answers.UserDisplayName: zooby
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-31T21:12:01.643Z
Answers.Text: No, you don't need to unroll the input sequence as long as you're building the computation graph as you go (which all modern ML packages will automatically do). Unrolling a fixed-length input sequence lets you (rather, the ML library) optimize the backpropagation, so it's definitely useful if you have the option, but it isn't a necessity
Answers.UserDisplayName: velocirabbit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-31T21:27:45.107Z
Answers.Text: Are you not saying that the RNN is unrolled behind-the-scenes?
Answers.UserDisplayName: zooby
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-31T22:10:33.053Z
Answers.Text: Not in the sense that memory space is reserved for it, no. Rather, the graph is built step-by-step and memory is allocated at each step.
Answers.UserDisplayName: velocirabbit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-31T01:38:06.970Z
Answers.LastActivityDate: 2018-07-31T01:38:06.970Z
Answers.UserDisplayName: velocirabbit
Answers.Body: <p>recurrent-neural-networks act on a sequence of inputs , that does not need to be only a time-sequence  for example consider a sequence of characters like a passage or a book.once trained on a sequence of inputs , you could predict the previous and next values of an input-vector at a certain time step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-31T17:16:05.743Z
Answers.LastActivityDate: 2018-05-31T17:16:05.743Z
Answers.UserDisplayName: thecomplexitytheorist
Body: <p>In an RNN to train it, you need to roll it out, and enter in the history of inputs and the history of expected outcomes. </p>
  
  <p>This doesn't seem like a realistic picture of the brain since this would require, for example, for the brain to store a perfect history of every sense that comes in to it for many time-steps. </p>
  
  <p>So is there an alternative to RNNs that doesn't require this history? Perhaps storing differences or something? Or storing some accumulator?</p>
  
  <p>Perhaps there is a way to calculate with RNNs that doesn't require keeping hold of this history?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-01T08:47:44.900Z
Comments.Text: This question should be migrated to: https://datascience.stackexchange.com/.
Comments.UserDisplayName: JahKnows
Tags: <recurrent-neural-networks>
Title: Is there an alternative to RNNs that doesn't require knowing input history?
UserDisplayName: zooby
ThreadText: # Is there an alternative to RNNs that doesn't require knowing input history?
  
  (Asked by: username_0 on 2018-05-18)
  
  In an RNN to train it, you need to roll it out, and enter in the history of inputs and the history of expected outcomes.
  
  This doesn't seem like a realistic picture of the brain since this would require, for example, for the brain to store a perfect history of every sense that comes in to it for many time-steps.
  
  So is there an alternative to RNNs that doesn't require this history? Perhaps storing differences or something? Or storing some accumulator?
  
  Perhaps there is a way to calculate with RNNs that doesn't require keeping hold of this history?
  
  **Tags:** recurrent-neural-networks
  
  > **Comments:**
  > **username_1**: This question should be migrated to: https://datascience.stackexchange.com/.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-07-31. Score: 1
  
  You don't necessarily need to roll out the inputs to an RNN; doing so makes it easier to optimize computation (if the sequence length is the same each batch), but it's not a necessity. Furthermore, RNNs (and, incidentally, the brain) doesn't necessarily remember the input history as is; rather, the history is encoded via the RNN's cell state (or states, in the case of LSTMs and other RNN cell architectures with multiple states). Neural Turing Machines (NMTs) and Differential neural computers expand on that concept by also using a larger "memory" storage (in the form of a matrix).
  
  > **Comments:**
  > **username_0**: I don't think that's true. To train an RNN you definitely need to roll it out. To use a pre-trained RNN you don't.  
  > **username_2**: No, you don't need to unroll the input sequence as long as you're building the computation graph as you go (which all modern ML packages will automatically do). Unrolling a fixed-length input sequence lets you (rather, the ML library) optimize the backpropagation, so it's definitely useful if you have the option, but it isn't a necessity  
  > **username_0**: Are you not saying that the RNN is unrolled behind-the-scenes?  
  > **username_2**: Not in the sense that memory space is reserved for it, no. Rather, the graph is built step-by-step and memory is allocated at each step.  
  
  ---
  
  ### Answer by username_3 on 2018-05-31. Score: 0
  
  recurrent-neural-networks act on a sequence of inputs , that does not need to be only a time-sequence for example consider a sequence of characters like a passage or a book.once trained on a sequence of inputs , you could predict the previous and next values of an input-vector at a certain time step.
  
  
  ---
  
  