Answers.Body: <p>Transformers can be trained with parallelization. So you might have 100 cores training the transformer for an hour (100 total hours of CPU time), whereas the LSTM has one core training for ten hours (10 total hours of CPU time). Even though the LSTM takes longer, it is less expensive to train.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-08T12:34:44.107Z
Answers.Text: This is what I also had as a final conclusion. So what this means is that if we would train a transformer for 10 hours of CPU time, or a LSTM for 10 hours of CPU time, results might be the same? And that if we invest more time in researching LSTMs we might even find them to be better (performance wise) & less compute expensive?
Answers.UserDisplayName: Tibo Geysen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-08T13:00:33.460Z
Answers.Text: Right, although to be honest I'm not familiar with the comparison of LSTMs and Transformers in terms of total training time—not sure if they're exactly the same or if one is better
Answers.UserDisplayName: Simon Alford
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-07T23:07:38.797Z
Answers.LastActivityDate: 2020-07-07T23:07:38.797Z
Answers.UserDisplayName: Simon Alford
Body: <p>In his paper introducing SHA-RNN (<a href="https://arxiv.org/pdf/1911.11423.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1911.11423.pdf</a>) Stephen Merity states that neglecting one direction of research (in this case LSTMs) over another (transformers) merily because the SOTA in transformers are due to using more computing power is not the way to go.</p>
  <p>I agree that finding neat tricks in AI/ML is equally (if not more) important than just throwing more computing power at the problem. However I am a little bit confused.</p>
  <p>The main difference (since they both use attention units) between his SHA-RNN and transformers seem to be the fact that SHA-RNN uses LSTMs to &quot;encode the position of words&quot;, where transformers do position encoding by using cosine and sine functions.</p>
  <p>My confusion comes from the fact that LSTMs need to be handled sequentially and thus they cannot use this large advantage of GPUs, being able to compute things in parallel, whilst transformers can. Wouldn't this mean that (assuming LSTMs and positional encoding are able to acquire the same results), training using LSTMs would take longer than transformers and thus need more computing power, thus defeating the initial puporse of this paper? Or am I misinterpreting this?</p>
  <p>Basically my question comes down to &quot;Why would an SHA-RNN be less computationally expensive than a transformer?&quot;</p>
  
Tags: <recurrent-neural-networks><long-short-term-memory><transformer><attention>
Title: What is the big fuzz about SHA-RNN versus Transformers?
UserDisplayName: Tibo Geysen
ThreadText: # What is the big fuzz about SHA-RNN versus Transformers?
  
  (Asked by: username_0 on 2020-07-07)
  
  In his paper introducing SHA-RNN (<https://arxiv.org/pdf/1911.11423.pdf>) Stephen Merity states that neglecting one direction of research (in this case LSTMs) over another (transformers) merily because the SOTA in transformers are due to using more computing power is not the way to go.
  
  I agree that finding neat tricks in AI/ML is equally (if not more) important than just throwing more computing power at the problem. However I am a little bit confused.
  
  The main difference (since they both use attention units) between his SHA-RNN and transformers seem to be the fact that SHA-RNN uses LSTMs to "encode the position of words", where transformers do position encoding by using cosine and sine functions.
  
  My confusion comes from the fact that LSTMs need to be handled sequentially and thus they cannot use this large advantage of GPUs, being able to compute things in parallel, whilst transformers can. Wouldn't this mean that (assuming LSTMs and positional encoding are able to acquire the same results), training using LSTMs would take longer than transformers and thus need more computing power, thus defeating the initial puporse of this paper? Or am I misinterpreting this?
  
  Basically my question comes down to "Why would an SHA-RNN be less computationally expensive than a transformer?"
  
  **Tags:** recurrent-neural-networks, long-short-term-memory, transformer, attention
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-07. Score: 1
  
  Transformers can be trained with parallelization. So you might have 100 cores training the transformer for an hour (100 total hours of CPU time), whereas the LSTM has one core training for ten hours (10 total hours of CPU time). Even though the LSTM takes longer, it is less expensive to train.
  
  > **Comments:**
  > **username_0**: This is what I also had as a final conclusion. So what this means is that if we would train a transformer for 10 hours of CPU time, or a LSTM for 10 hours of CPU time, results might be the same? And that if we invest more time in researching LSTMs we might even find them to be better (performance wise) & less compute expensive?  
  > **username_1**: Right, although to be honest I'm not familiar with the comparison of LSTMs and Transformers in terms of total training time—not sure if they're exactly the same or if one is better  
  
  ---
  
  