Body: <p>The main advantages of the self-attention mechanism are:</p>
  <ul>
  <li>Ability to capture long-range dependencies</li>
  <li>Ease to parallelize on GPU or TPU</li>
  </ul>
  <p>However, I wonder why the same goals cannot be achieved by <em>global depthwise convolution</em> (with the kernel size equal to the length of the input sequence) with a comparable amount of flops.</p>
  <p><strong>Note</strong>:</p>
  <p>In the following, I am comparing against the original architecture from the paper <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention Is All You Need</a>.</p>
  <p><strong>Idea</strong>:</p>
  <p>Consider the depthwise convolution of size <span class="math-container">$L$</span> with circular padding:
  <span class="math-container">$$
  y_{t,c} = W_{t^{'},c} x_{t^{'} + t, c}
  $$</span>
  Here, <span class="math-container">$x$</span> is the input signal and <span class="math-container">$y$</span> is the output signal,
  <span class="math-container">$t$</span> is the position in the sequence, and <span class="math-container">$c$</span> is the channel index.
  Since the convolution is depthwise the given output channel depends on the unique input channel (we would like to have linear complexity in the dimension of the embedding vector).</p>
  <p>After a single convolution, one definitely would not have any interactions between the tokens in the sequence.</p>
  <p>However, a two-layer convolutional network with these tokens is able to capture long-range pair-wise interactions:
  <span class="math-container">$$
  x_{t,c}^{(2)} = W_{t^{''},c}^{(2)} \sigma(W_{t^{'},c}^{(1)} x_{t^{'} + t, c}^{(0)})
  $$</span></p>
  <p>And by stacking a not very large number of these layers (like 12 or 24) one can model interactions between tokens in the sequence of arbitrary complexity.</p>
  <p><strong>Comparison of complexity</strong>:</p>
  <p>The asymptotic complexity of both approaches seems to be the same.</p>
  <ul>
  <li><p>Attention: <span class="math-container">$O (L^2 d)$</span></p>
  </li>
  <li><p>Depthwise convolution: <span class="math-container">$O (L^2 d)$</span></p>
  </li>
  </ul>
  <p>However, dot product attention seems to be a rather intuitive and biologically motivated operation that is crucial for sequence problems.</p>
  <p>Has this question been studied in the literature or discussed somewhere before?</p>
  <p><strong>EDIT</strong></p>
  <p>De-facto global depthwise convolution is used in <a href="https://arxiv.org/abs/2105.01601" rel="nofollow noreferrer">MLP-Mixer</a>. One stage performs convolution with global receptive field (of the size of feature map), and other operation is pointwise convolution with <code>kernel_size=1</code>.</p>
  <p><a href="https://i.sstatic.net/sSE0H.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/sSE0H.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-05T16:44:18.190Z
Comments.Text: This is an old question, but can you please provide a source that describes this "global depthwise convolution"?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-05T17:31:29.213Z
Comments.Text: @nbro ok, I'll add information
Comments.UserDisplayName: spiridon_the_sun_rotator
Tags: <reference-request><attention><convolution><sequence-modeling>
Title: Couldn't the self-attention mechanism be replaced with a global depth-wise convolution?
UserDisplayName: spiridon_the_sun_rotator
ThreadText: # Couldn't the self-attention mechanism be replaced with a global depth-wise convolution?
  
  (Asked by: username_0 on 2021-08-14)
  
  The main advantages of the self-attention mechanism are:
  
  * Ability to capture long-range dependencies
  * Ease to parallelize on GPU or TPU
  
  However, I wonder why the same goals cannot be achieved by *global depthwise convolution* (with the kernel size equal to the length of the input sequence) with a comparable amount of flops.
  
  **Note**:
  
  In the following, I am comparing against the original architecture from the paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762).
  
  **Idea**:
  
  Consider the depthwise convolution of size $L$ with circular padding:
  $$
  y\_{t,c} = W\_{t^{'},c} x\_{t^{'} + t, c}
  $$
  Here, $x$ is the input signal and $y$ is the output signal,
  $t$ is the position in the sequence, and $c$ is the channel index.
  Since the convolution is depthwise the given output channel depends on the unique input channel (we would like to have linear complexity in the dimension of the embedding vector).
  
  After a single convolution, one definitely would not have any interactions between the tokens in the sequence.
  
  However, a two-layer convolutional network with these tokens is able to capture long-range pair-wise interactions:
  $$
  x\_{t,c}^{(2)} = W\_{t^{''},c}^{(2)} \sigma(W\_{t^{'},c}^{(1)} x\_{t^{'} + t, c}^{(0)})
  $$
  
  And by stacking a not very large number of these layers (like 12 or 24) one can model interactions between tokens in the sequence of arbitrary complexity.
  
  **Comparison of complexity**:
  
  The asymptotic complexity of both approaches seems to be the same.
  
  * Attention: $O (L^2 d)$
  * Depthwise convolution: $O (L^2 d)$
  
  However, dot product attention seems to be a rather intuitive and biologically motivated operation that is crucial for sequence problems.
  
  Has this question been studied in the literature or discussed somewhere before?
  
  **EDIT**
  
  De-facto global depthwise convolution is used in [MLP-Mixer](https://arxiv.org/abs/2105.01601). One stage performs convolution with global receptive field (of the size of feature map), and other operation is pointwise convolution with `kernel_size=1`.
  
  [![enter image description here](https://i.sstatic.net/sSE0H.png)](https://i.sstatic.net/sSE0H.png)
  
  **Tags:** reference-request, attention, convolution, sequence-modeling
  
  > **Comments:**
  > **username_1**: This is an old question, but can you please provide a source that describes this "global depthwise convolution"?  
  > **username_0**: @username_1 ok, I'll add information  
  
  ---
  
  ## 0 answers
  
  