Answers.Body: <p>Yes, you are correct and it was one of original motivations, which inspired the invention of the <code>Attention</code> mechanism in <code>seq2seq</code> problems <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1706.03762.pdf</a>.</p>
  <p>There is a quote from this paper:</p>
  <blockquote>
  <p>Recurrent models typically factor computation along the symbol
  positions of the input and output sequences. Aligning the positions to
  steps in computation time, they generate a sequence of hidden states <span class="math-container">$h_t$</span>,
  as a function of the previous hidden state <span class="math-container">$h_{t−1}$</span> and the input for
  position <span class="math-container">$t$</span>. This inherently sequential nature precludes parallelization
  within training examples, which becomes critical at longer sequence
  lengths, as memory constraints limit batching across examples.</p>
  </blockquote>
  <p>On the other hand, <code>Transformer</code> architectures have a loot loom for parallelization, because they take the whole sequence at once, and multiple heads can be executed in parallel.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-25T03:41:37.523Z
Answers.Text: Thanks man! And one more thing, apart from the `H` (Hiddne State MAtrix) here, everything else is shared. `WhX, WhY, F` are all shared. Right?
Answers.UserDisplayName: Deshwal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-25T04:37:20.243Z
Answers.Text: And also, All of these 4 matrices have the same dimensionality which we define by (Number of Neurons) right?
Answers.UserDisplayName: Deshwal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-25T07:46:11.000Z
Answers.Text: `WhX, WhY, F` are shared. In fact you have only one cell - only the inputs `x_t` and `h_t` change. However, I would not call `F` a matrix - it is usually an activation function. About the dimensinality - no, in general setting. The dimension of the output `y_t` doesn't need to be equal to `x_t` or `h_t`. And `x_t` and `h_t` can have different dimensionality as well. This `WhX` and `WhY` are of different size.
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-25T09:51:28.707Z
Answers.Text: So how can you decide the size of `WhX` and `WhY`? Also, if `F` is an activation function, then there are only 3 matrices not 4 right? But I thought there are 4 matrices one of which has all 0s in starting and is responsible for the state change? Right? [See this link](https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks) 4 matrices and 2 biases.
Answers.UserDisplayName: Deshwal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-25T13:14:10.733Z
Answers.Text: In your link I see only 3 matrices. There is a matrix `Wxh` which mutliplies the input and is added to form the hidde state, matrix `Whh` from hidden state to hidden state, and the matrix `Why` from hidden state to output `y`. `x`, `h`, `y` can be of different dimensionality. Thus, `Wxh` is a `dim_x * dim_h` matrix, `Whh` is a `dim_h * dim_h` and so on.
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-25T14:27:48.877Z
Answers.Text: Actually I was taking the Hidden Vector as a matrix too. Thats why I got confused. Thanks anyways for helping so long.
Answers.UserDisplayName: Deshwal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-24T20:42:34.457Z
Answers.LastActivityDate: 2021-01-24T20:42:34.457Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>You take any blog or any example and all they tell you about is the given picture below.</p>
  <p><a href="https://i.sstatic.net/hyb25.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hyb25.png" alt="enter image description here" /></a></p>
  <p>It has 4 different matrices and 3 of whose weights are shared. So, I'm wondering how is this achieved in practice?</p>
  <p>Please correct me:</p>
  <p>I think the first word &quot;hello&quot; goes in as a one-hot encoded form and changes the Hidden matrix. And then after it, &quot;world&quot; goes and gets multiplied and then changes the matrix again and so on. What people make it look like is that all of the words going are in Parallel. It can't be the case because the Hidden matrix is dependent on the previous word and without changing the matric, you can not pass the current word. Please correct if my idea is wrong but I think the execution is in sequential order.</p>
  
Tags: <neural-networks><deep-learning><natural-language-processing><recurrent-neural-networks><long-short-term-memory>
Title: Is the working of RNNs, LSTM and GRU sequential or parallel?
UserDisplayName: Deshwal
ThreadText: # Is the working of RNNs, LSTM and GRU sequential or parallel?
  
  (Asked by: username_0 on 2021-01-24)
  
  You take any blog or any example and all they tell you about is the given picture below.
  
  [![enter image description here](https://i.sstatic.net/hyb25.png)](https://i.sstatic.net/hyb25.png)
  
  It has 4 different matrices and 3 of whose weights are shared. So, I'm wondering how is this achieved in practice?
  
  Please correct me:
  
  I think the first word "hello" goes in as a one-hot encoded form and changes the Hidden matrix. And then after it, "world" goes and gets multiplied and then changes the matrix again and so on. What people make it look like is that all of the words going are in Parallel. It can't be the case because the Hidden matrix is dependent on the previous word and without changing the matric, you can not pass the current word. Please correct if my idea is wrong but I think the execution is in sequential order.
  
  **Tags:** neural-networks, deep-learning, natural-language-processing, recurrent-neural-networks, long-short-term-memory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-24. Score: 2 | Accepted answer
  
  Yes, you are correct and it was one of original motivations, which inspired the invention of the `Attention` mechanism in `seq2seq` problems <https://arxiv.org/pdf/1706.03762.pdf>.
  
  There is a quote from this paper:
  
  > Recurrent models typically factor computation along the symbol
  > positions of the input and output sequences. Aligning the positions to
  > steps in computation time, they generate a sequence of hidden states $h\_t$,
  > as a function of the previous hidden state $h\_{t−1}$ and the input for
  > position $t$. This inherently sequential nature precludes parallelization
  > within training examples, which becomes critical at longer sequence
  > lengths, as memory constraints limit batching across examples.
  
  On the other hand, `Transformer` architectures have a loot loom for parallelization, because they take the whole sequence at once, and multiple heads can be executed in parallel.
  
  > **Comments:**
  > **username_0**: Thanks man! And one more thing, apart from the `H` (Hiddne State MAtrix) here, everything else is shared. `WhX, WhY, F` are all shared. Right?  
  > **username_0**: And also, All of these 4 matrices have the same dimensionality which we define by (Number of Neurons) right?  
  > **username_1**: `WhX, WhY, F` are shared. In fact you have only one cell - only the inputs `x_t` and `h_t` change. However, I would not call `F` a matrix - it is usually an activation function. About the dimensinality - no, in general setting. The dimension of the output `y_t` doesn't need to be equal to `x_t` or `h_t`. And `x_t` and `h_t` can have different dimensionality as well. This `WhX` and `WhY` are of different size.  
  > **username_0**: So how can you decide the size of `WhX` and `WhY`? Also, if `F` is an activation function, then there are only 3 matrices not 4 right? But I thought there are 4 matrices one of which has all 0s in starting and is responsible for the state change? Right? [See this link](https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks) 4 matrices and 2 biases.  
  > **username_1**: In your link I see only 3 matrices. There is a matrix `Wxh` which mutliplies the input and is added to form the hidde state, matrix `Whh` from hidden state to hidden state, and the matrix `Why` from hidden state to output `y`. `x`, `h`, `y` can be of different dimensionality. Thus, `Wxh` is a `dim_x * dim_h` matrix, `Whh` is a `dim_h * dim_h` and so on.  
  > **username_0**: Actually I was taking the Hidden Vector as a matrix too. Thats why I got confused. Thanks anyways for helping so long.  
  
  ---
  
  