Answers.Body: <p>Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is that <em>attention</em> should be used when a model is capable of <em>learning</em>, or <em>focusing on</em> local vs global patterns in the data we use for training. And with &quot;should be used&quot; I simply refer to the fact that everyone like to feel eligible to write &quot;Hey, we used attention!&quot; simply because of the hype generated by the introduction of transformers by Vaswani et al.</p>
  <p>Said that, I think up to this point the best expression to describe attention is:</p>
  <p><strong>A specific type of architecture</strong></p>
  <p>What do I mean by this: Vaswani et al. introduced the expression <em>attention</em> in the paper you cite with a new whole machine learning architecture, namely the transformers. In the paper, attention is used to refer to a specific set of layers, similarly we call <em><a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">residual blocks</a></em> or <em><a href="https://arxiv.org/pdf/1608.06993.pdf" rel="nofollow noreferrer">dense blocks</a></em> specific type of layers combinations that were introduced for convolutional neural networks. For me the is no difference at all between attention and the two above mentioned examples. The confusion around the use of this expression in my opinion arose from the fact that Vaswani et al. put a lot of emphasis on the final purpose of the new proposed model, i.e. capturing local similarities within sentences in machine translation.</p>
  <p>One last consideration why I think that architecture is the best label for attention is that it include also type of attentions that are completely different from the multi-head attention module introduced by Vaswani et al, like architectures that leverage <a href="https://arxiv.org/pdf/1802.09070.pdf" rel="nofollow noreferrer">attention maps</a>. Mathematically, attention maps and the multi-head attention module share nothing but the name, still, because conceptually they seems to fulfill the same purpose, we call them both attention, with the consequence that to avoid confusion, one should always refer to a specific paper when talking about attention.</p>
  <p><a href="https://i.sstatic.net/Tg9yj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Tg9yj.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/utRM2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/utRM2.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-19T14:28:59.080Z
Answers.Text: This answer is good (I think, although I am not an expert in this topic), but there's one thing that may not be clear from your explanation. If "attention" is best described as a "layer" or "specific architecture" (submodule of a neural network), then, **conceptually**, what exactly does it do? (I think this was the question). You say "because conceptually they seems to fulfill the same purpose". What is the purpose then? That's one thing that is not emphasized enough in your answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-19T14:29:10.473Z
Answers.Text: I think that, before the transformer, attention was already gaining "attention" (sorry for the pun) within the machine translation community (see neural machine translation, for instance), from what I remember. The transformer paper certainly made the name "attention" and all related ideas even more famous, but I think it's important to note/emphasize that the idea of attention was already in the air before the transformer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-19T14:29:13.500Z
Answers.Text: You should probably also clarify this sentence "For me the is no difference at all between attention and the two above mentioned examples" by saying that there's no difference in terms of role in a neural network (I think that's what you mean), i.e. that "attention" is just a layer or a module of a neural network, in the same way that "skip connections" are just specific types of layers or submodules.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-19T14:05:07.430Z
Answers.LastActivityDate: 2021-10-19T14:05:07.430Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>The answer above is very concise but I will try to give an ELI5 example. I also agree with @nbro that attention does not exclusively mean transformer architecture.</p>
  <hr />
  <h3>Before attention</h3>
  <p>What is the height of the youngest female child of the father of your mother's first cousin? That query is convoluted, depends on your good memory of your family tree's relationships. It looks like this:</p>
  <p><span class="math-container">$Q_{RNN} = f(Mother(first\_cousin(father(yongest\_female(height))))\ \ (1)$</span></p>
  <p>What you see in (1) is how Recurrent Neural Networks function, and it is obvious that their performance is inversely correlated with the length of the input.</p>
  <h3>After attention</h3>
  <p>A more efficient way to query the height of that person is by giving it a name - Aligna. Now, whenever you need to get characteristics of that person you don't have to remember it's &quot;chained&quot; relationships backwards from the person that is closest to you. This new approach would look like this:</p>
  <p><span class="math-container">$Q_{attentive_RNN} = f(Aligna(height))\ \ (2)$</span></p>
  <p>The latter is what essentially attention pooling does. <em>Attention establishes a direct link with all input steps (e.g. words in an input sentence) so that it can independently pay attention to the input (or set of inputs) that will generate a successful output/prediction.</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-20T12:45:24.267Z
Answers.LastActivityDate: 2021-10-20T12:45:24.267Z
Answers.UserDisplayName: hH1sG0n3
Body: <p>I heard from many people about the paper titled <strong><a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">Attention Is All You Need</a></strong> by <em>Ashish Vaswani et al</em>.</p>
  <p>What actually does the &quot;attention&quot; do in simple terms? Is it a function, property, or some other thing?</p>
  
Tags: <neural-networks><papers><transformer><attention>
Title: In layman terms, what does "attention" do in a transformer?
UserDisplayName: hanugm
ThreadText: # In layman terms, what does "attention" do in a transformer?
  
  (Asked by: username_0 on 2021-10-19)
  
  I heard from many people about the paper titled **[Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf)** by *Ashish Vaswani et al*.
  
  What actually does the "attention" do in simple terms? Is it a function, property, or some other thing?
  
  **Tags:** neural-networks, papers, transformer, attention
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-10-19. Score: 4
  
  Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is that *attention* should be used when a model is capable of *learning*, or *focusing on* local vs global patterns in the data we use for training. And with "should be used" I simply refer to the fact that everyone like to feel eligible to write "Hey, we used attention!" simply because of the hype generated by the introduction of transformers by Vaswani et al.
  
  Said that, I think up to this point the best expression to describe attention is:
  
  **A specific type of architecture**
  
  What do I mean by this: Vaswani et al. introduced the expression *attention* in the paper you cite with a new whole machine learning architecture, namely the transformers. In the paper, attention is used to refer to a specific set of layers, similarly we call *[residual blocks](https://arxiv.org/pdf/1512.03385.pdf)* or *[dense blocks](https://arxiv.org/pdf/1608.06993.pdf)* specific type of layers combinations that were introduced for convolutional neural networks. For me the is no difference at all between attention and the two above mentioned examples. The confusion around the use of this expression in my opinion arose from the fact that Vaswani et al. put a lot of emphasis on the final purpose of the new proposed model, i.e. capturing local similarities within sentences in machine translation.
  
  One last consideration why I think that architecture is the best label for attention is that it include also type of attentions that are completely different from the multi-head attention module introduced by Vaswani et al, like architectures that leverage [attention maps](https://arxiv.org/pdf/1802.09070.pdf). Mathematically, attention maps and the multi-head attention module share nothing but the name, still, because conceptually they seems to fulfill the same purpose, we call them both attention, with the consequence that to avoid confusion, one should always refer to a specific paper when talking about attention.
  
  [![enter image description here](https://i.sstatic.net/Tg9yj.png)](https://i.sstatic.net/Tg9yj.png)
  
  [![enter image description here](https://i.sstatic.net/utRM2.png)](https://i.sstatic.net/utRM2.png)
  
  > **Comments:**
  > **username_2**: This answer is good (I think, although I am not an expert in this topic), but there's one thing that may not be clear from your explanation. If "attention" is best described as a "layer" or "specific architecture" (submodule of a neural network), then, **conceptually**, what exactly does it do? (I think this was the question). You say "because conceptually they seems to fulfill the same purpose". What is the purpose then? That's one thing that is not emphasized enough in your answer.  
  > **username_2**: I think that, before the transformer, attention was already gaining "attention" (sorry for the pun) within the machine translation community (see neural machine translation, for instance), from what I remember. The transformer paper certainly made the name "attention" and all related ideas even more famous, but I think it's important to note/emphasize that the idea of attention was already in the air before the transformer.  
  > **username_2**: You should probably also clarify this sentence "For me the is no difference at all between attention and the two above mentioned examples" by saying that there's no difference in terms of role in a neural network (I think that's what you mean), i.e. that "attention" is just a layer or a module of a neural network, in the same way that "skip connections" are just specific types of layers or submodules.  
  
  ---
  
  ### Answer by username_3 on 2021-10-20. Score: 0
  
  The answer above is very concise but I will try to give an ELI5 example. I also agree with @username_2 that attention does not exclusively mean transformer architecture.
  
  ---
  
  ### Before attention
  
  What is the height of the youngest female child of the father of your mother's first cousin? That query is convoluted, depends on your good memory of your family tree's relationships. It looks like this:
  
  $Q\_{RNN} = f(Mother(first\\_cousin(father(yongest\\_female(height))))\ \ (1)$
  
  What you see in (1) is how Recurrent Neural Networks function, and it is obvious that their performance is inversely correlated with the length of the input.
  
  ### After attention
  
  A more efficient way to query the height of that person is by giving it a name - Aligna. Now, whenever you need to get characteristics of that person you don't have to remember it's "chained" relationships backwards from the person that is closest to you. This new approach would look like this:
  
  $Q\_{attentive\_RNN} = f(Aligna(height))\ \ (2)$
  
  The latter is what essentially attention pooling does. *Attention establishes a direct link with all input steps (e.g. words in an input sentence) so that it can independently pay attention to the input (or set of inputs) that will generate a successful output/prediction.*
  
  
  ---
  
  