<p>As per &quot;Attention is all you need&quot; etc., positional encoding is added to the embedded word vector input. My knee-jerk reaction is that this would muddle the &quot;signal&quot; of the word vector. Since word order is not preserved, this additive noise, for instance, could make a word look like a different word (in a different position): <span class="math-container">$w_a + p_a = x = w_b + p_b$</span>.</p>
<p>Is performance (from shorter input dimension) the main reason to add the positional information versus concatenating, or is adding theoretically sound?</p>
<p><strong>Edit:</strong> I think a source of confusion in all of this is the assumption of whether the model adopts a pre-trained word embedder, like Word2Vec, or trains their embedder from scratch. Adding positional encoding to a pre-trained embedder might throw off the attention dot product?</p>

CC BY-SA 4.0
2024-01-04T14:54:45.747Z
You may want to do some calculations to understand just how little the positional embeddings move the word embeddings to see if that helps you intuitively.
David Hoelzer
CC BY-SA 4.0
2024-01-04T15:02:39.630Z
Range from PE would be [-1,1] which I imagine is also the word embed range, no?
SuaveSouris
CC BY-SA 4.0
2024-01-04T19:27:48.420Z
I'd say elegance is the main reason in conjunction with the evidence that the addition works well: If you'd concatenate you'd need another dimensionality reducing transformation to make subsequent residual connections work. Also check [this](https://ai.stackexchange.com/q/35990/52211) related question, I like the orthogonality argument in the [first](https://ai.stackexchange.com/a/43213/52211) answer, second answer is my own attempt of explanation ;)
Chillston
CC BY-SA 4.0
2024-01-06T20:28:44.013Z
How is this question different from [Why are embeddings added, not concatenated?"](https://ai.stackexchange.com/q/35990/61494) that @Chillston linked above?
Eponymous
<attention><word-embedding><positional-encoding>
Concatenating the positional encoding
SuaveSouris
# Concatenating the positional encoding

(Asked by: username_0 on 2024-01-04)

As per "Attention is all you need" etc., positional encoding is added to the embedded word vector input. My knee-jerk reaction is that this would muddle the "signal" of the word vector. Since word order is not preserved, this additive noise, for instance, could make a word look like a different word (in a different position): $w\_a + p\_a = x = w\_b + p\_b$.

Is performance (from shorter input dimension) the main reason to add the positional information versus concatenating, or is adding theoretically sound?

**Edit:** I think a source of confusion in all of this is the assumption of whether the model adopts a pre-trained word embedder, like Word2Vec, or trains their embedder from scratch. Adding positional encoding to a pre-trained embedder might throw off the attention dot product?

**Tags:** attention, word-embedding, positional-encoding

> **Comments:**
> **username_1**: You may want to do some calculations to understand just how little the positional embeddings move the word embeddings to see if that helps you intuitively.  
> **username_0**: Range from PE would be [-1,1] which I imagine is also the word embed range, no?  
> **username_2**: I'd say elegance is the main reason in conjunction with the evidence that the addition works well: If you'd concatenate you'd need another dimensionality reducing transformation to make subsequent residual connections work. Also check [this](https://ai.stackexchange.com/q/35990/52211) related question, I like the orthogonality argument in the [first](https://ai.stackexchange.com/a/43213/52211) answer, second answer is my own attempt of explanation ;)  
> **username_3**: How is this question different from [Why are embeddings added, not concatenated?"](https://ai.stackexchange.com/q/35990/61494) that @username_2 linked above?  

---

## 0 answers

