Answers.Body: <p>Note that the architecture of the encoder (and decoder) uses residual connections that add the original input to the output of the self attention layer. Thus, the information from the positional information will be preserved in the deeper layers.</p>
  <p>The encoder also uses a position-wise mlp after the self-attention, after which we also apply the residual connection. For more details (and figures) on the encoder architecture see <a href="https://pi-tau.github.io/posts/transformer/#encoder-block" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-23T21:41:33.703Z
Answers.LastActivityDate: 2023-07-23T21:41:33.703Z
Answers.UserDisplayName: pi-tau
Answers.Body: <p>Attention mechanism solves this problem by allowing the decoder to “look-back” at the encoder's hidden states based on its current state. This allows the decoder to extract only relevant information about the input tokens at each decoding, thus learning more complicated dependencies between the input and the output.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-31T12:31:20.447Z
Answers.Text: But what about fully encoder architectures? Like ViTs in computer vision?
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-31T13:05:40.240Z
Answers.Text: In addition, as far as I remember, usually, decoder receives information from the last layer of encoder. There are multiple of them, hence the question remains.
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-31T11:31:09.920Z
Answers.LastActivityDate: 2021-10-31T11:31:09.920Z
Answers.UserDisplayName: robert4johnson
Body: <p>Transformer architecture (without position embedding) is by the very construction equivariant to the <strong>permutation</strong> of tokens. Given query <span class="math-container">$Q \in \mathbb{R}^{n \times d}$</span> and keys <span class="math-container">$K \in \mathbb{R}^{n \times d}$</span> and some permutation matrix <span class="math-container">$P \in \mathbb{R}^{n \times n}$</span>, one has:
  <span class="math-container">$$
  Q \rightarrow P Q, K \rightarrow P K 
  $$</span>
  <span class="math-container">$$\begin{align}
  A &amp;= 
  \text{softmax} \left(\frac{Q K^T}{\sqrt{d}} \right) \\ &amp;\rightarrow 
  \text{softmax} \left(\frac{R Q K^T R^T}{\sqrt{d}} \right) \\
  &amp;= 
  R \ \text{softmax} \left(\frac{Q K^T}{\sqrt{d}} \right) R^T \\
  &amp;= R A R^T
  \end{align}$$</span>
  Without the <strong>positional embedding</strong> (learned of fixed), that breaks permutation symmetry to translational there is no notion of location. And with the positional embedding one introduces a notion - token <span class="math-container">$x$</span> is on the <span class="math-container">$k$</span>-th position.</p>
  <p>However, I wonder, whether this notion makes sense after the first self-attention layer.</p>
  <p>The operation of producing the output (multiplication of attention by the value)
  <span class="math-container">$$
  x_{out} = A V
  $$</span>
  outputs some weighted sum of <strong>all</strong> tokens in the sequence, the token at the <span class="math-container">$k$</span>-th position now has information from all the sequence.</p>
  <p>So, I wonder, whether the notion of <em>position</em> (absolute or relative) still makes sense in this case?</p>
  <p><a href="https://i.sstatic.net/Y4VbC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Y4VbC.png" alt="enter image description here" /></a></p>
  <p>My guess is, that since Transformers involve skip connections, these transfer the notion of the location to the next layers. But this depends on the relative magnitude between the activation of the given self-attention layer and the skip-connections</p>
  
Tags: <transformer><attention><positional-encoding>
Title: Is there a notion of location in Transformer architecture in subsequent self-attention layers?
UserDisplayName: spiridon_the_sun_rotator
ThreadText: # Is there a notion of location in Transformer architecture in subsequent self-attention layers?
  
  (Asked by: username_0 on 2021-10-31)
  
  Transformer architecture (without position embedding) is by the very construction equivariant to the **permutation** of tokens. Given query $Q \in \mathbb{R}^{n \times d}$ and keys $K \in \mathbb{R}^{n \times d}$ and some permutation matrix $P \in \mathbb{R}^{n \times n}$, one has:
  $$
  Q \rightarrow P Q, K \rightarrow P K
  $$
  $$\begin{align}
  A &=
  \text{softmax} \left(\frac{Q K^T}{\sqrt{d}} \right) \\ &\rightarrow
  \text{softmax} \left(\frac{R Q K^T R^T}{\sqrt{d}} \right) \\
  &=
  R \ \text{softmax} \left(\frac{Q K^T}{\sqrt{d}} \right) R^T \\
  &= R A R^T
  \end{align}$$
  Without the **positional embedding** (learned of fixed), that breaks permutation symmetry to translational there is no notion of location. And with the positional embedding one introduces a notion - token $x$ is on the $k$-th position.
  
  However, I wonder, whether this notion makes sense after the first self-attention layer.
  
  The operation of producing the output (multiplication of attention by the value)
  $$
  x\_{out} = A V
  $$
  outputs some weighted sum of **all** tokens in the sequence, the token at the $k$-th position now has information from all the sequence.
  
  So, I wonder, whether the notion of *position* (absolute or relative) still makes sense in this case?
  
  [![enter image description here](https://i.sstatic.net/Y4VbC.png)](https://i.sstatic.net/Y4VbC.png)
  
  My guess is, that since Transformers involve skip connections, these transfer the notion of the location to the next layers. But this depends on the relative magnitude between the activation of the given self-attention layer and the skip-connections
  
  **Tags:** transformer, attention, positional-encoding
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-07-23. Score: 1
  
  Note that the architecture of the encoder (and decoder) uses residual connections that add the original input to the output of the self attention layer. Thus, the information from the positional information will be preserved in the deeper layers.
  
  The encoder also uses a position-wise mlp after the self-attention, after which we also apply the residual connection. For more details (and figures) on the encoder architecture see [here](https://pi-tau.github.io/posts/transformer/#encoder-block).
  
  
  ---
  
  ### Answer by username_2 on 2021-10-31. Score: 0
  
  Attention mechanism solves this problem by allowing the decoder to “look-back” at the encoder's hidden states based on its current state. This allows the decoder to extract only relevant information about the input tokens at each decoding, thus learning more complicated dependencies between the input and the output.
  
  > **Comments:**
  > **username_0**: But what about fully encoder architectures? Like ViTs in computer vision?  
  > **username_0**: In addition, as far as I remember, usually, decoder receives information from the last layer of encoder. There are multiple of them, hence the question remains.  
  
  ---
  
  