Answers.Body: <p>It is true that when using local attention with a window of size 5, the &quot;receptive field&quot; is the same as a CNN with kernel size 5 (or two CNN layers with kernel size 3). However, there is a key difference in how the learned weights are applied to the inputs.</p>
  <p><strong>In a CNN</strong>, the values of the many convolutional kernels are learned, but once learned, <strong>the kernels are static</strong>. In other words, at every position in the input (whether it be a 1D signal or 2D image), the dot product between the inputs within the window and the same CNN kernels is taken, and then a non-linear function applied.</p>
  <p><strong>With attention, the Query/Key/Value matrices additionally allow context to be taken into account</strong>. Instead of taking the dot-product of the input region with a set of fixed kernels, the additional matrices are effectively used to <strong>dynamically compute a new set of kernels</strong> for each position. &quot;Attention&quot; basically figures out for each convolution, which inputs are important (which inputs the network should &quot;pay attention&quot; to) by computing higher-valued weights using Q, K, and V.</p>
  <p>I highly recommend reading a breakdown of the original &quot;Attention is All You Need&quot; paper such as this blog post: <a href="https://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">https://jalammar.github.io/illustrated-transformer/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-12T06:56:29.480Z
Answers.Text: Yeah, I've read this famous blog. Seems like the recently introduced involution operation https://arxiv.org/abs/2103.06255 is something in between attention and convolution operation.
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-10T21:45:49.180Z
Answers.LastActivityDate: 2021-07-10T21:45:49.180Z
Answers.UserDisplayName: TimD1
Body: <p>Transformer architectures, based on the self-attention mechanism, have achieved outstanding performance in a variety of applications.</p>
  <p>The main advantage of this approach is that the given token can interact with any token in the input sequence and extract global information since the first layer, whereas CNN has to stack multiple convolutional or pooling layers in order to achieve a <em>receptive</em> field, that would involve the whole input sequence.</p>
  <p>By <em>receptive</em> field I mean the number of timestamps from the input signal on which does the output depend. For example, for sequence of two <code>Conv1D</code> with <code>kernel_size=3</code> receptive field is 5. And in transformer the output of the first blocks depends on the whole sequence.</p>
  <p>However, this comes at large computational and memory cost in the vanilla formulation:
  <span class="math-container">$$
  O(L^2)
  $$</span>
  where <span class="math-container">$L$</span> is the length of the sequence.</p>
  <p>There have been proposed various mechanisms, that try to reduce this amount of computation:</p>
  <ul>
  <li>Random attention</li>
  <li>Window (Local attention)</li>
  <li>Global attention</li>
  </ul>
  <p>All these forms of attention are illustrated below:</p>
  <p><a href="https://i.sstatic.net/6Rf7D.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6Rf7D.png" alt="enter image description here" /></a></p>
  <p>And one can combine different of these approaches as in the <a href="https://arxiv.org/abs/2007.14062" rel="nofollow noreferrer">Big Bird paper</a></p>
  <p>My question is about local attention, attending only to the tokens in the fixed neighborhood of size <span class="math-container">$K$</span>.
  By doing so, one reduces the number of operations to:
  <span class="math-container">$$
  O(L K)
  $$</span>
  However, now it is local as the ordinary convolution, and global receptive field will be achieved only via stacking many layers.</p>
  <p>Are there any advantages of Local self-attention against CNN, or it can be beneficial only in combination with other forms of attention?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-11T23:51:11.047Z
Comments.Text: To answer this VERY important question one should decide what to keep constant in a comparison of a 1d convnet and a transformer (e.g., in a language modelling application): number of parameters, FLOPS, dollars spent for training, dollars spent for inference? I would probably go for a combination of the latter two. The final result would be very much influenced by how much we know about "best architectural choices" for the two classes of models (much more for transformers?).
Comments.UserDisplayName: user4422
Tags: <convolutional-neural-networks><attention><convolution>
Title: Are there any advantages of the local attention against convolutions?
UserDisplayName: spiridon_the_sun_rotator
ThreadText: # Are there any advantages of the local attention against convolutions?
  
  (Asked by: username_0 on 2021-07-09)
  
  Transformer architectures, based on the self-attention mechanism, have achieved outstanding performance in a variety of applications.
  
  The main advantage of this approach is that the given token can interact with any token in the input sequence and extract global information since the first layer, whereas CNN has to stack multiple convolutional or pooling layers in order to achieve a *receptive* field, that would involve the whole input sequence.
  
  By *receptive* field I mean the number of timestamps from the input signal on which does the output depend. For example, for sequence of two `Conv1D` with `kernel_size=3` receptive field is 5. And in transformer the output of the first blocks depends on the whole sequence.
  
  However, this comes at large computational and memory cost in the vanilla formulation:
  $$
  O(L^2)
  $$
  where $L$ is the length of the sequence.
  
  There have been proposed various mechanisms, that try to reduce this amount of computation:
  
  * Random attention
  * Window (Local attention)
  * Global attention
  
  All these forms of attention are illustrated below:
  
  [![enter image description here](https://i.sstatic.net/6Rf7D.png)](https://i.sstatic.net/6Rf7D.png)
  
  And one can combine different of these approaches as in the [Big Bird paper](https://arxiv.org/abs/2007.14062)
  
  My question is about local attention, attending only to the tokens in the fixed neighborhood of size $K$.
  By doing so, one reduces the number of operations to:
  $$
  O(L K)
  $$
  However, now it is local as the ordinary convolution, and global receptive field will be achieved only via stacking many layers.
  
  Are there any advantages of Local self-attention against CNN, or it can be beneficial only in combination with other forms of attention?
  
  **Tags:** convolutional-neural-networks, attention, convolution
  
  > **Comments:**
  > **username_1**: To answer this VERY important question one should decide what to keep constant in a comparison of a 1d convnet and a transformer (e.g., in a language modelling application): number of parameters, FLOPS, dollars spent for training, dollars spent for inference? I would probably go for a combination of the latter two. The final result would be very much influenced by how much we know about "best architectural choices" for the two classes of models (much more for transformers?).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-07-10. Score: 4
  
  It is true that when using local attention with a window of size 5, the "receptive field" is the same as a CNN with kernel size 5 (or two CNN layers with kernel size 3). However, there is a key difference in how the learned weights are applied to the inputs.
  
  **In a CNN**, the values of the many convolutional kernels are learned, but once learned, **the kernels are static**. In other words, at every position in the input (whether it be a 1D signal or 2D image), the dot product between the inputs within the window and the same CNN kernels is taken, and then a non-linear function applied.
  
  **With attention, the Query/Key/Value matrices additionally allow context to be taken into account**. Instead of taking the dot-product of the input region with a set of fixed kernels, the additional matrices are effectively used to **dynamically compute a new set of kernels** for each position. "Attention" basically figures out for each convolution, which inputs are important (which inputs the network should "pay attention" to) by computing higher-valued weights using Q, K, and V.
  
  I highly recommend reading a breakdown of the original "Attention is All You Need" paper such as this blog post: <https://jalammar.github.io/illustrated-transformer/>
  
  > **Comments:**
  > **username_0**: Yeah, I've read this famous blog. Seems like the recently introduced involution operation https://arxiv.org/abs/2103.06255 is something in between attention and convolution operation.  
  
  ---
  
  