Answers.Body: <p>If you go through the <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">main introductory paper of the transformer</a> (&quot;Attention is all you need&quot;), you can find the comparison of the model with other state-of-the-art machine translation method:</p>
  <p><a href="https://i.sstatic.net/o4SRo.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/o4SRo.png" alt="enter image description here" /></a></p>
  <p>For example, <a href="https://arxiv.org/pdf/1606.04199.pdf" rel="nofollow noreferrer">Deep-Att + PosUnk</a> is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the transformer with self-attention is <span class="math-container">$2.3 \cdot 10^{19}$</span> (FLOPs) and <span class="math-container">$1.0 \cdot 10^{20}$</span> (FLOPs) for the &quot;Deep-Att + PosUnk&quot; method (the transformer is 4 times faster) on &quot;WMT14 English-to-French&quot; dataset.</p>
  <p>Please note that the BLEU is a crucial factor here (not merely training cost). Hence, you can see the BLEU‌ value of the transformer superior to the <a href="https://arxiv.org/pdf/1610.10099.pdf" rel="nofollow noreferrer">ByteNet</a> (Neural Machine Translation in Linear Time). Although the ByteNet has not adopted the RNN, you can find the comparison of the ByteNet with other &quot;RNN + Attention&quot; methods in its original paper:</p>
  <p><a href="https://i.sstatic.net/abdit.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/abdit.png" alt="enter image description here" /></a></p>
  <p>Hence, by transitivity property of the BLEU score, you can find that the transformer has already outperformed other &quot;RNN‌ + Attention&quot; methods in terms of the BLEU score (please check their performance on &quot;WMT14&quot; dataset).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-12T23:10:41.200Z
Answers.Text: Thanks @OmG! Your answer is what I was looking for! Thank you. By the way, what if the dataset is limited?
Answers.UserDisplayName: milad aghajohari
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-13T12:28:36.777Z
Answers.Text: My pleasure @miladaghajohari . Due to the translation task's high complexity, the required model's VC dimension will be increased. Hence, based on the computational learning theory, the required sample complexity to train the model will be increased as well. Therefore, a limitted dataset is unlikely to be suitable for the task.
Answers.UserDisplayName: OmG
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-05T09:19:48.303Z
Answers.LastActivityDate: 2020-10-05T09:44:26.683Z
Answers.LastEditDate: 2020-10-05T09:44:26.683Z
Answers.UserDisplayName: OmG
Body: <p>I am wondering what is believed to be the reason for superiority of transformer?</p>
  <p>I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. However, as far as I know, you can use attention also with RNN  architectures as in the famous paper attention is introduced(<a href="https://arxiv.org/pdf/1409.0473" rel="nofollow noreferrer">here</a>)).</p>
  <p>I am wondering whether the only reason for the superiority of transformers is because they can be highly parallelized and trained on much more data?</p>
  <p>Is there any experiment comparing transformers and RNN+attention trained on the exact same amount of data comparing the two?</p>
  
Tags: <natural-language-processing><recurrent-neural-networks><long-short-term-memory><transformer><attention>
Title: Any comparison between transformer and RNN+Attention on the same dataset?
UserDisplayName: milad aghajohari
ThreadText: # Any comparison between transformer and RNN+Attention on the same dataset?
  
  (Asked by: username_0 on 2020-10-04)
  
  I am wondering what is believed to be the reason for superiority of transformer?
  
  I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. However, as far as I know, you can use attention also with RNN architectures as in the famous paper attention is introduced([here](https://arxiv.org/pdf/1409.0473))).
  
  I am wondering whether the only reason for the superiority of transformers is because they can be highly parallelized and trained on much more data?
  
  Is there any experiment comparing transformers and RNN+attention trained on the exact same amount of data comparing the two?
  
  **Tags:** natural-language-processing, recurrent-neural-networks, long-short-term-memory, transformer, attention
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-05. Score: 3 | Accepted answer
  
  If you go through the [main introductory paper of the transformer](https://arxiv.org/pdf/1706.03762.pdf) ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation method:
  
  [![enter image description here](https://i.sstatic.net/o4SRo.png)](https://i.sstatic.net/o4SRo.png)
  
  For example, [Deep-Att + PosUnk](https://arxiv.org/pdf/1606.04199.pdf) is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the transformer with self-attention is $2.3 \cdot 10^{19}$ (FLOPs) and $1.0 \cdot 10^{20}$ (FLOPs) for the "Deep-Att + PosUnk" method (the transformer is 4 times faster) on "WMT14 English-to-French" dataset.
  
  Please note that the BLEU is a crucial factor here (not merely training cost). Hence, you can see the BLEU‌ value of the transformer superior to the [ByteNet](https://arxiv.org/pdf/1610.10099.pdf) (Neural Machine Translation in Linear Time). Although the ByteNet has not adopted the RNN, you can find the comparison of the ByteNet with other "RNN + Attention" methods in its original paper:
  
  [![enter image description here](https://i.sstatic.net/abdit.png)](https://i.sstatic.net/abdit.png)
  
  Hence, by transitivity property of the BLEU score, you can find that the transformer has already outperformed other "RNN‌ + Attention" methods in terms of the BLEU score (please check their performance on "WMT14" dataset).
  
  > **Comments:**
  > **username_0**: Thanks @OmG! Your answer is what I was looking for! Thank you. By the way, what if the dataset is limited?  
  > **username_1**: My pleasure @miladaghajohari . Due to the translation task's high complexity, the required model's VC dimension will be increased. Hence, based on the computational learning theory, the required sample complexity to train the model will be increased as well. Therefore, a limitted dataset is unlikely to be suitable for the task.  
  
  ---
  
  