Answers.Body: <p>I understand the confusion. Although transformers are autoregressive (they predict something based on past information), they are not recurrent (do not have hidden states).</p>
  <p>In fact, think of a transformer as a multi-layer perceptron. Each input neuron is fully connected to every neuron in the first hidden layer. If you would swap the places of the first and the second input neuron (including their associated weights), all the connections/weights stay the same, but the order of the neurons is different.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-11T07:42:37.773Z
Answers.LastActivityDate: 2023-05-11T07:42:37.773Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>It is clear that word positions are essential for the meaning of a sentence, and so are essential when feeding a sentence (= sequence of words) as a matrix of word embedding vectors into a transformer. I also have understood roughly how positions are encoded, but what I did not understand in the very begining is <strong>why</strong> just creating a matrix consisting of a number of word embedding vectors (as columns) with the columns in the same order as the words in the sentence does not suffice. A matrix with permutated columns obviously would &quot;mean&quot; something different - and sometimes nothing at all - like a pixel matrix would change its &quot;meaning&quot; when we permutated some pixel columns. Is there an intuitive explanation why position encoding vectors have to be added to the word embedding vectors. Why and how would the position information (which is still present in the input matrix) get lost otherwise?</p>
  <p><strike>(I have learned in the meanwhile that transformers are in general <strong>not</strong> permutation invariant, but that there are transformers that <strong>are</strong>: <a href="https://arxiv.org/abs/1810.00825" rel="nofollow noreferrer">set transformers</a>.)</strike></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-11T07:09:16.230Z
Comments.Text: Transformers are permutation invariant, the positional encoding added (by summing or concatenation) does not change that fact. can you rephrase your question?
Comments.UserDisplayName: N. Kiefer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-11T10:22:00.207Z
Comments.Text: @N.Kiefer: I was misled by Lee's paper on set transformers which seems to suggest that general (non-set) transformers were **not** permutation invariant. So if the truth is that general transformers **are** in general permutation invariant, this would perfectly answer my question.
Comments.UserDisplayName: Hans-Peter Stricker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-11T10:24:39.243Z
Comments.Text: @N.Kiefer: Convolutional networks in turn are really not permutation invariant, right? That's where the analogy between pixel matrices (as inputs to CNNs) and word-embedding matrices (as inputs to Transformers, which are fully connected) fails, right?
Comments.UserDisplayName: Hans-Peter Stricker
Tags: <transformer><word-embedding><positional-encoding>
Title: What is the intuition behind position-encoding?
UserDisplayName: Hans-Peter Stricker
ThreadText: # What is the intuition behind position-encoding?
  
  (Asked by: username_0 on 2023-05-10)
  
  It is clear that word positions are essential for the meaning of a sentence, and so are essential when feeding a sentence (= sequence of words) as a matrix of word embedding vectors into a transformer. I also have understood roughly how positions are encoded, but what I did not understand in the very begining is **why** just creating a matrix consisting of a number of word embedding vectors (as columns) with the columns in the same order as the words in the sentence does not suffice. A matrix with permutated columns obviously would "mean" something different - and sometimes nothing at all - like a pixel matrix would change its "meaning" when we permutated some pixel columns. Is there an intuitive explanation why position encoding vectors have to be added to the word embedding vectors. Why and how would the position information (which is still present in the input matrix) get lost otherwise?
  
  (I have learned in the meanwhile that transformers are in general **not** permutation invariant, but that there are transformers that **are**: [set transformers](https://arxiv.org/abs/1810.00825).)
  
  **Tags:** transformer, word-embedding, positional-encoding
  
  > **Comments:**
  > **username_1**: Transformers are permutation invariant, the positional encoding added (by summing or concatenation) does not change that fact. can you rephrase your question?  
  > **username_0**: @N.username_1: I was misled by Lee's paper on set transformers which seems to suggest that general (non-set) transformers were **not** permutation invariant. So if the truth is that general transformers **are** in general permutation invariant, this would perfectly answer my question.  
  > **username_0**: @N.username_1: Convolutional networks in turn are really not permutation invariant, right? That's where the analogy between pixel matrices (as inputs to CNNs) and word-embedding matrices (as inputs to Transformers, which are fully connected) fails, right?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-11. Score: 3
  
  I understand the confusion. Although transformers are autoregressive (they predict something based on past information), they are not recurrent (do not have hidden states).
  
  In fact, think of a transformer as a multi-layer perceptron. Each input neuron is fully connected to every neuron in the first hidden layer. If you would swap the places of the first and the second input neuron (including their associated weights), all the connections/weights stay the same, but the order of the neurons is different.
  
  
  ---
  
  