Answers.Body: <p>You are talking about model parallelism. But, that's not the reason RNNs/LSTMs are not in vogue.</p>
  <p>Imagine your ability to read the first line of a page and going on reading and still making connections to the first line until the end of the page.</p>
  <p>Can RNNs/LSTMs do that? No.
  Can Attention (i.e. Transformers) do it? Yes.</p>
  <p>The reason is simple Attention is effectively an affinity matrix between each and every input that goes into a network. So, it is able to do that. We have a huge memory overload but hey, we want the performance.</p>
  <p>In case of RNNs/LSTMs, the cells have to do this heavy-lifting, there is only a set amount of information that can be contained in them. That's why you have to forget gate to control information retained.</p>
  <p>Nevertheless, your thought is correct but that's not the reason for Attention to be in vogue. But, your thought has negative ramifications when we see how to implement it. Also, nevertheless the computation will be still sequential since you can't process input (n + 1) without input n. Local parallelization is possible but not global.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T07:28:42.060Z
Answers.Text: Thank you, Abhishek, for your answer! Regarding the implementation of parallelization, could you specify in a bit more detail why it would be more complex to do so in batch-dimension (in the case of RNNs/LSTMs) compared to how parallelization is done in the case of Attention?
Answers.UserDisplayName: zock
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T07:56:30.120Z
Answers.Text: https://leimao.github.io/blog/Data-Parallelism-vs-Model-Paralelism/
Answers.UserDisplayName: Abhishek Verma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T08:35:30.480Z
Answers.Text: Hi Abhishek, thanks for the link but I cannot fully understand your point. I don't know if Model parallelism is often used for Attention or not, but data parallelism seems simpler than model parallelism to me. On huggingface pre-trained models, they also make use of data parallelism and support sending sub-batches to each GPU/CPU device.
Answers.UserDisplayName: zock
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T08:41:47.010Z
Answers.Text: That's exactly the point. What you were talking about is model parallelism which is hard because it is difficult to consolidate back the weights calculated on different workers.
Answers.UserDisplayName: Abhishek Verma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T08:48:33.980Z
Answers.Text: hmm, please correct me if I am wrong, I think I was talking about using RNNs/LSTMs with data parallelism, not model parallelism, isn't it?
Answers.UserDisplayName: zock
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T09:28:52.990Z
Answers.Text: In data parallelism, the model is replicated. You are not talking about model replication. You are saying you want to break the model computations, that is model parallelism. Breaking model into various workers.
Answers.UserDisplayName: Abhishek Verma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T17:27:56.503Z
Answers.LastActivityDate: 2021-04-06T17:27:56.503Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly sequential. That is, the computation of the next tokens depends on the result of previous tokens, thus, RNNs are losing to Attention in terms of speed.</p>
  <p>Even though I fully agree that RNNs are sequential as stated above, I think they are still parallelizable by splitting the mini-batch into sub-batches and each of these sub-batches is processed independently by a dedicated thread. For example, a training batch of size 32 can be split into 4 sub-batches of size 8; 4 threads process 4 sub-batches independently. That way, RNNs/LSTMs are parallelizable and this is not a disadvantage compared to Attention.</p>
  <p>Is my thought correct?</p>
  
Tags: <machine-learning><deep-learning><recurrent-neural-networks><long-short-term-memory><attention>
Title: Do RNNs/LSTMs really need to be sequential?
UserDisplayName: zock
ThreadText: # Do RNNs/LSTMs really need to be sequential?
  
  (Asked by: username_0 on 2021-04-06)
  
  There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly sequential. That is, the computation of the next tokens depends on the result of previous tokens, thus, RNNs are losing to Attention in terms of speed.
  
  Even though I fully agree that RNNs are sequential as stated above, I think they are still parallelizable by splitting the mini-batch into sub-batches and each of these sub-batches is processed independently by a dedicated thread. For example, a training batch of size 32 can be split into 4 sub-batches of size 8; 4 threads process 4 sub-batches independently. That way, RNNs/LSTMs are parallelizable and this is not a disadvantage compared to Attention.
  
  Is my thought correct?
  
  **Tags:** machine-learning, deep-learning, recurrent-neural-networks, long-short-term-memory, attention
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-06. Score: 2 | Accepted answer
  
  You are talking about model parallelism. But, that's not the reason RNNs/LSTMs are not in vogue.
  
  Imagine your ability to read the first line of a page and going on reading and still making connections to the first line until the end of the page.
  
  Can RNNs/LSTMs do that? No.
  Can Attention (i.e. Transformers) do it? Yes.
  
  The reason is simple Attention is effectively an affinity matrix between each and every input that goes into a network. So, it is able to do that. We have a huge memory overload but hey, we want the performance.
  
  In case of RNNs/LSTMs, the cells have to do this heavy-lifting, there is only a set amount of information that can be contained in them. That's why you have to forget gate to control information retained.
  
  Nevertheless, your thought is correct but that's not the reason for Attention to be in vogue. But, your thought has negative ramifications when we see how to implement it. Also, nevertheless the computation will be still sequential since you can't process input (n + 1) without input n. Local parallelization is possible but not global.
  
  > **Comments:**
  > **username_0**: Thank you, Abhishek, for your answer! Regarding the implementation of parallelization, could you specify in a bit more detail why it would be more complex to do so in batch-dimension (in the case of RNNs/LSTMs) compared to how parallelization is done in the case of Attention?  
  > **username_1**: https://leimao.github.io/blog/Data-Parallelism-vs-Model-Paralelism/  
  > **username_0**: Hi Abhishek, thanks for the link but I cannot fully understand your point. I don't know if Model parallelism is often used for Attention or not, but data parallelism seems simpler than model parallelism to me. On huggingface pre-trained models, they also make use of data parallelism and support sending sub-batches to each GPU/CPU device.  
  > **username_1**: That's exactly the point. What you were talking about is model parallelism which is hard because it is difficult to consolidate back the weights calculated on different workers.  
  > **username_0**: hmm, please correct me if I am wrong, I think I was talking about using RNNs/LSTMs with data parallelism, not model parallelism, isn't it?  
  > **username_1**: In data parallelism, the model is replicated. You are not talking about model replication. You are saying you want to break the model computations, that is model parallelism. Breaking model into various workers.  
  
  ---
  
  