Answers.Body: <p>The positional embedding values in transformers are fixed from the start, they are calculated only once before the start of training (or stored/reused from years ago in memory).</p>
  <p>So there is only very specific word embedding vector that can &quot;confuse&quot; the combination of word + position vector values.</p>
  <p>Suppose embedding for a word is so coincidentally unfortunate as to fall into this category. Then the prediction accuracy of the transformer will decrease as a result of this unfortunate property. This causes readjustment of the word embedding (since position embeddings are fixed and unchangable).</p>
  <p>Thus, training will naturally eliminate any existing ambiguity of word + position embeddings between different word vectors.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-30T05:00:33.207Z
Answers.LastActivityDate: 2024-04-30T05:00:33.207Z
Answers.UserDisplayName: James
Answers.Body: <p>This would be incredibly difficult just due to the size of these embedding vectors and the fact that we are performing 2D rotations (many in parallel). The larger this embedding space the more the vectors will be spread out. So the main argument is not that it is impossible but that it is incredibly unlikely. The odds that any 2 vectors are a series of 2D rotations apart is low and becomes even lower in large embedding spaces.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-07T19:48:59.003Z
Answers.LastActivityDate: 2024-05-07T19:48:59.003Z
Answers.UserDisplayName: naston
Body: <p>In Transformers, we add the positional embedding with the word embedding and then process it. But, what if the sum of the embeddings become same for different words at different position of a sentence? How would the model differentiate it?</p>
  
Tags: <transformer><word-embedding>
Title: What if the sum of word embedding and positional embedding becomes same for different words?
UserDisplayName: Aakash Roy
ThreadText: # What if the sum of word embedding and positional embedding becomes same for different words?
  
  (Asked by: username_0 on 2024-04-27)
  
  In Transformers, we add the positional embedding with the word embedding and then process it. But, what if the sum of the embeddings become same for different words at different position of a sentence? How would the model differentiate it?
  
  **Tags:** transformer, word-embedding
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-04-30. Score: 0
  
  The positional embedding values in transformers are fixed from the start, they are calculated only once before the start of training (or stored/reused from years ago in memory).
  
  So there is only very specific word embedding vector that can "confuse" the combination of word + position vector values.
  
  Suppose embedding for a word is so coincidentally unfortunate as to fall into this category. Then the prediction accuracy of the transformer will decrease as a result of this unfortunate property. This causes readjustment of the word embedding (since position embeddings are fixed and unchangable).
  
  Thus, training will naturally eliminate any existing ambiguity of word + position embeddings between different word vectors.
  
  
  ---
  
  ### Answer by username_2 on 2024-05-07. Score: 0
  
  This would be incredibly difficult just due to the size of these embedding vectors and the fact that we are performing 2D rotations (many in parallel). The larger this embedding space the more the vectors will be spread out. So the main argument is not that it is impossible but that it is incredibly unlikely. The odds that any 2 vectors are a series of 2D rotations apart is low and becomes even lower in large embedding spaces.
  
  
  ---
  
  