Answers.Body: <p>The original transformer is a feedforward neural network (FFNN)-based architecture that <em>makes use of</em> an attention mechanism. So, this is the difference: an attention mechanism (in particular, a <em>self-attention</em> operation) is used by the transformer, which is not just this attention mechanism, but it's an encoder-decoder architecture, which makes use of other techniques too: for example, <em>positional encoding</em> and <em>layer normalization</em>. In other words, the transformer is the model, while the attention is a technique used by the model.</p>
  <p>The paper that introduced the transformer <a href="https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="noreferrer">Attention Is All You Need</a> (2017, NIPS) contains a diagram of the transformer and the attention block (i.e. the part of the transformer that does this attention operation).</p>
  <p>Here's the diagram of the transformer.</p>
  <p><a href="https://i.sstatic.net/RZacm.png" rel="noreferrer"><img src="https://i.sstatic.net/RZacm.png" alt="enter image description here" /></a></p>
  <p>Here's the picture of the attention mechanism (as you can see from the diagram above, the transformer used the <em>multi-head attention</em> on the right).</p>
  <p><a href="https://i.sstatic.net/xJIS3.png" rel="noreferrer"><img src="https://i.sstatic.net/xJIS3.png" alt="enter image description here" /></a></p>
  <p>One thing to keep in mind is that the idea of attention is not novel to the transformer, given that similar ideas had already been used in previous works and models, for example, <a href="https://arxiv.org/pdf/1409.0473.pdf" rel="noreferrer">here</a>, although the specific attention mechanisms are different.</p>
  <p>Of course, you should read the mentioned paper for more details about the transformer, the attention mechanism and the diagrams above.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-06T22:48:14.540Z
Answers.Text: Could you reply to this question, please?https://datascience.stackexchange.com/questions/124020/how-can-we-use-a-transfomer-model-with-new-data-if-we-still-dont-have-the-outpu/124022#124022
Answers.UserDisplayName: skan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-26T11:41:49.143Z
Answers.LastActivityDate: 2021-07-26T11:41:49.143Z
Answers.UserDisplayName: nbro
Body: <p>Having been studying computer vision for a while, I still cannot understand what the difference between a transformer and attention is?</p>
  
Tags: <computer-vision><comparison><transformer><attention>
Title: In Computer Vision, what is the difference between a transformer and attention?
UserDisplayName: novice
ThreadText: # In Computer Vision, what is the difference between a transformer and attention?
  
  (Asked by: username_0 on 2021-07-25)
  
  Having been studying computer vision for a while, I still cannot understand what the difference between a transformer and attention is?
  
  **Tags:** computer-vision, comparison, transformer, attention
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-26. Score: 11 | Accepted answer
  
  The original transformer is a feedforward neural network (FFNN)-based architecture that *makes use of* an attention mechanism. So, this is the difference: an attention mechanism (in particular, a *self-attention* operation) is used by the transformer, which is not just this attention mechanism, but it's an encoder-decoder architecture, which makes use of other techniques too: for example, *positional encoding* and *layer normalization*. In other words, the transformer is the model, while the attention is a technique used by the model.
  
  The paper that introduced the transformer [Attention Is All You Need](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) (2017, NIPS) contains a diagram of the transformer and the attention block (i.e. the part of the transformer that does this attention operation).
  
  Here's the diagram of the transformer.
  
  [![enter image description here](https://i.sstatic.net/RZacm.png)](https://i.sstatic.net/RZacm.png)
  
  Here's the picture of the attention mechanism (as you can see from the diagram above, the transformer used the *multi-head attention* on the right).
  
  [![enter image description here](https://i.sstatic.net/xJIS3.png)](https://i.sstatic.net/xJIS3.png)
  
  One thing to keep in mind is that the idea of attention is not novel to the transformer, given that similar ideas had already been used in previous works and models, for example, [here](https://arxiv.org/pdf/1409.0473.pdf), although the specific attention mechanisms are different.
  
  Of course, you should read the mentioned paper for more details about the transformer, the attention mechanism and the diagrams above.
  
  > **Comments:**
  > **username_2**: Could you reply to this question, please?https://datascience.stackexchange.com/questions/124020/how-can-we-use-a-transfomer-model-with-new-data-if-we-still-dont-have-the-outpu/124022#124022  
  
  ---
  
  