Answers.Body: <p>The recurrent connections are also used during the forward pass. Take a look, for example, at the following equations that compute (during the forward pass) the hidden state <span class="math-container">$h_t$</span> of an LSTM layer.</p>
  <p><span class="math-container">\begin{align}
  f_t &amp;= \sigma_g(W_{f} x_t + \color{red}{U_{f}} h_{t-1} + b_f) \\
  i_t &amp;= \sigma_g(W_{i} x_t + \color{red}{U_{i}} h_{t-1} + b_i) \\
  o_t &amp;= \sigma_g(W_{o} x_t + \color{red}{U_{o}} h_{t-1} + b_o) \\
  \tilde{c}_t &amp;= \sigma_c(W_{c} x_t + \color{red}{U_{c}} h_{t-1} + b_c) \\
  c_t &amp;= f_t \circ c_{t-1} + i_t \circ \tilde{c}_t \\
  h_t &amp;= o_t \circ \sigma_h(c_t)
  \end{align}</span></p>
  <p>The letters in <span class="math-container">$\color{red}{\text{red}}$</span> represent the matrices associated with the <span class="math-container">$\color{red}{\text{recurrent connections}}$</span>, so we use them in the forward pass. Of course, before training, these connections are useless, in the sense that, if we initialize them randomly, they do not initially provide any context. However, during the backward pass, we update them based on our (labeled) data, so, over time, they should capture the temporal/recurrent relations of the sequences.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-17T11:36:59.733Z
Answers.LastActivityDate: 2021-12-17T11:46:08.260Z
Answers.LastEditDate: 2021-12-17T11:46:08.260Z
Answers.UserDisplayName: nbro
Body: <p>Consider the following paragraph taken from <a href="https://www.deeplearningbook.org/contents/rnn.html" rel="nofollow noreferrer">chapter 10: Sequence Modeling: Recurrent and Recursive Nets</a> of the textbook named <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer">Deep Learning</a> by <em>Ian Goodfellow et al</em> mentioning the connections of RNN to go backward in time.</p>
  <blockquote>
  <p>For the simplicity of exposition, we refer to RNNs as operating on a
  sequence that contains vectors <span class="math-container">$x^{(t)}$</span> with the time step index <span class="math-container">$t$</span>
  ranging from 1 to <span class="math-container">$\tau$</span>. In practice, recurrent networks usually
  operate on minibatches of such sequences, with a diﬀerent sequence
  lengthτfor each member of the minibatch. We have omitted the minibatch
  indices to simplify notation. Moreover, the time step index need not
  literally refer to the passage of time in the real world. Sometimes it
  refers only to the position in the sequence. RNNs may also be applied
  in two dimensions across spatial data such as images, and even when
  applied to data involving time, <strong>the network may have connections
  that go backward in time, provided that the entire sequence is
  observed before it is provided to the network.</strong></p>
  </blockquote>
  <p>The paragraph says that the RNN can go back in time if and only if the entire sequence is provided. So, I am suspecting that it happens <em>only</em> during the backpropagation/backward pass.</p>
  <p>Am I true? Or is it possible for an RNN to use those connections while forward pass also?</p>
  
Tags: <deep-learning><recurrent-neural-networks>
Title: When does an RNN use the connections that help in going backward in time?
UserDisplayName: hanugm
ThreadText: # When does an RNN use the connections that help in going backward in time?
  
  (Asked by: username_0 on 2021-12-17)
  
  Consider the following paragraph taken from [chapter 10: Sequence Modeling: Recurrent and Recursive Nets](https://www.deeplearningbook.org/contents/rnn.html) of the textbook named [Deep Learning](https://www.deeplearningbook.org/) by *Ian Goodfellow et al* mentioning the connections of RNN to go backward in time.
  
  > For the simplicity of exposition, we refer to RNNs as operating on a
  > sequence that contains vectors $x^{(t)}$ with the time step index $t$
  > ranging from 1 to $\tau$. In practice, recurrent networks usually
  > operate on minibatches of such sequences, with a diﬀerent sequence
  > lengthτfor each member of the minibatch. We have omitted the minibatch
  > indices to simplify notation. Moreover, the time step index need not
  > literally refer to the passage of time in the real world. Sometimes it
  > refers only to the position in the sequence. RNNs may also be applied
  > in two dimensions across spatial data such as images, and even when
  > applied to data involving time, **the network may have connections
  > that go backward in time, provided that the entire sequence is
  > observed before it is provided to the network.**
  
  The paragraph says that the RNN can go back in time if and only if the entire sequence is provided. So, I am suspecting that it happens *only* during the backpropagation/backward pass.
  
  Am I true? Or is it possible for an RNN to use those connections while forward pass also?
  
  **Tags:** deep-learning, recurrent-neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-17. Score: 2
  
  The recurrent connections are also used during the forward pass. Take a look, for example, at the following equations that compute (during the forward pass) the hidden state $h\_t$ of an LSTM layer.
  
  \begin{align}
  f\_t &= \sigma\_g(W\_{f} x\_t + \color{red}{U\_{f}} h\_{t-1} + b\_f) \\
  i\_t &= \sigma\_g(W\_{i} x\_t + \color{red}{U\_{i}} h\_{t-1} + b\_i) \\
  o\_t &= \sigma\_g(W\_{o} x\_t + \color{red}{U\_{o}} h\_{t-1} + b\_o) \\
  \tilde{c}\_t &= \sigma\_c(W\_{c} x\_t + \color{red}{U\_{c}} h\_{t-1} + b\_c) \\
  c\_t &= f\_t \circ c\_{t-1} + i\_t \circ \tilde{c}\_t \\
  h\_t &= o\_t \circ \sigma\_h(c\_t)
  \end{align}
  
  The letters in $\color{red}{\text{red}}$ represent the matrices associated with the $\color{red}{\text{recurrent connections}}$, so we use them in the forward pass. Of course, before training, these connections are useless, in the sense that, if we initialize them randomly, they do not initially provide any context. However, during the backward pass, we update them based on our (labeled) data, so, over time, they should capture the temporal/recurrent relations of the sequences.
  
  
  ---
  
  