Answers.Body: <p>An influential predecessor paper is indeed the work on <a href="https://arxiv.org/pdf/1409.0473.pdf" rel="nofollow noreferrer">NEURAL MACHINE TRANSLATION
  BY JOINTLY LEARNING TO ALIGN AND TRANSLATE</a>. The paper outlines an attentional mechanism that is similar to the computation of the actual attentional weights in the transformer paper. See the paper and picture below for details.</p>
  <p><a href="https://i.sstatic.net/7i0JI.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7i0JI.jpg" alt="Model overview of attention mechanism" /></a></p>
  <p>When the authors of the paper visualize their attention weights they obtain the following results. The inputs into the attention computation are not called queries and keys yet but are conceptually similar.</p>
  <p><a href="https://i.sstatic.net/KJtI1.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KJtI1.jpg" alt="Attention weight visualization" /></a></p>
  <p>See this <a href="https://www.youtube.com/watch?v=EixI6t5oif0" rel="nofollow noreferrer">video</a> for a detailed explanation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-15T09:51:56.340Z
Answers.LastActivityDate: 2022-12-15T10:11:21.137Z
Answers.LastEditDate: 2022-12-15T10:11:21.137Z
Answers.UserDisplayName: Mariusmarten
Body: <p>I'm wondering what the origins of the transformer as proposed in <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">Attention Is All You Need</a> are. The paper itself provides some interesting pointers to the literature on self-attention such as:</p>
  <ol>
  <li><a href="https://aclanthology.org/D16-1244.pdf" rel="nofollow noreferrer">A Decomposable Attention Model for Natural Language Inference</a></li>
  <li><a href="https://openreview.net/pdf?id=BJC_jUqxe" rel="nofollow noreferrer">A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING</a></li>
  <li><a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE</a></li>
  </ol>
  <p>It seems like using attentional mechanisms was widespread (at least in combination with recurrent networks) and 'A Decomposable Attention Model for Natural Language Inference' paper from 2016 already conjectured that scaling attention might be feasible.
  Is it from this prior work 'only' an engineering leap? Or what additional papers at the time likely influenced the architecture?</p>
  
Tags: <natural-language-processing><transformer><attention>
Title: What is the most important predecessor of the transformer model?
UserDisplayName: Mariusmarten
ThreadText: # What is the most important predecessor of the transformer model?
  
  (Asked by: username_0 on 2022-08-02)
  
  I'm wondering what the origins of the transformer as proposed in [Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf) are. The paper itself provides some interesting pointers to the literature on self-attention such as:
  
  1. [A Decomposable Attention Model for Natural Language Inference](https://aclanthology.org/D16-1244.pdf)
  2. [A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING](https://openreview.net/pdf?id=BJC_jUqxe)
  3. [NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE](https://arxiv.org/pdf/1706.03762.pdf)
  
  It seems like using attentional mechanisms was widespread (at least in combination with recurrent networks) and 'A Decomposable Attention Model for Natural Language Inference' paper from 2016 already conjectured that scaling attention might be feasible.
  Is it from this prior work 'only' an engineering leap? Or what additional papers at the time likely influenced the architecture?
  
  **Tags:** natural-language-processing, transformer, attention
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-12-15. Score: 3 | Accepted answer
  
  An influential predecessor paper is indeed the work on [NEURAL MACHINE TRANSLATION
  BY JOINTLY LEARNING TO ALIGN AND TRANSLATE](https://arxiv.org/pdf/1409.0473.pdf). The paper outlines an attentional mechanism that is similar to the computation of the actual attentional weights in the transformer paper. See the paper and picture below for details.
  
  [![Model overview of attention mechanism](https://i.sstatic.net/7i0JI.jpg)](https://i.sstatic.net/7i0JI.jpg)
  
  When the authors of the paper visualize their attention weights they obtain the following results. The inputs into the attention computation are not called queries and keys yet but are conceptually similar.
  
  [![Attention weight visualization](https://i.sstatic.net/KJtI1.jpg)](https://i.sstatic.net/KJtI1.jpg)
  
  See this [video](https://www.youtube.com/watch?v=EixI6t5oif0) for a detailed explanation.
  
  
  ---
  
  