Body: <p>As per &quot;Attention is all you need&quot; etc., positional encoding is added to the embedded word vector input. My knee-jerk reaction is that this would muddle the &quot;signal&quot; of the word vector. Since word order is not preserved, this additive noise, for instance, could make a word look like a different word (in a different position): <span class="math-container">$w_a + p_a = x = w_b + p_b$</span>.</p>
  <p>Is performance (from shorter input dimension) the main reason to add the positional information versus concatenating, or is adding theoretically sound?</p>
  <p><strong>Edit:</strong> I think a source of confusion in all of this is the assumption of whether the model adopts a pre-trained word embedder, like Word2Vec, or trains their embedder from scratch. Adding positional encoding to a pre-trained embedder might throw off the attention dot product?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-04T14:54:45.747Z
Comments.Text: You may want to do some calculations to understand just how little the positional embeddings move the word embeddings to see if that helps you intuitively.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-04T15:02:39.630Z
Comments.Text: Range from PE would be [-1,1] which I imagine is also the word embed range, no?
Comments.UserDisplayName: SuaveSouris
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-04T19:27:48.420Z
Comments.Text: I'd say elegance is the main reason in conjunction with the evidence that the addition works well: If you'd concatenate you'd need another dimensionality reducing transformation to make subsequent residual connections work. Also check [this](https://ai.stackexchange.com/q/35990/52211) related question, I like the orthogonality argument in the [first](https://ai.stackexchange.com/a/43213/52211) answer, second answer is my own attempt of explanation ;)
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-06T20:28:44.013Z
Comments.Text: How is this question different from [Why are embeddings added, not concatenated?"](https://ai.stackexchange.com/q/35990/61494) that @Chillston linked above?
Comments.UserDisplayName: Eponymous
Tags: <attention><word-embedding><positional-encoding>
Title: Concatenating the positional encoding
UserDisplayName: SuaveSouris
ThreadText: # Concatenating the positional encoding
  
  (Asked by: username_0 on 2024-01-04)
  
  As per "Attention is all you need" etc., positional encoding is added to the embedded word vector input. My knee-jerk reaction is that this would muddle the "signal" of the word vector. Since word order is not preserved, this additive noise, for instance, could make a word look like a different word (in a different position): $w\_a + p\_a = x = w\_b + p\_b$.
  
  Is performance (from shorter input dimension) the main reason to add the positional information versus concatenating, or is adding theoretically sound?
  
  **Edit:** I think a source of confusion in all of this is the assumption of whether the model adopts a pre-trained word embedder, like Word2Vec, or trains their embedder from scratch. Adding positional encoding to a pre-trained embedder might throw off the attention dot product?
  
  **Tags:** attention, word-embedding, positional-encoding
  > **Comments:**
  > **username_1**: You may want to do some calculations to understand just how little the positional embeddings move the word embeddings to see if that helps you intuitively.  
  > **username_0**: Range from PE would be [-1,1] which I imagine is also the word embed range, no?  
  > **username_2**: I'd say elegance is the main reason in conjunction with the evidence that the addition works well: If you'd concatenate you'd need another dimensionality reducing transformation to make subsequent residual connections work. Also check [this](https://ai.stackexchange.com/q/35990/52211) related question, I like the orthogonality argument in the [first](https://ai.stackexchange.com/a/43213/52211) answer, second answer is my own attempt of explanation ;)  
  > **username_3**: How is this question different from [Why are embeddings added, not concatenated?"](https://ai.stackexchange.com/q/35990/61494) that @Chillston linked above?  
  
  ---
  
  ## 0 answers
  
  