Answers.Body: <p>Positional Encoding (PE) is a broad term encompassing various techniques meant to embed positional information within a matrix <span class="math-container">$x \in \mathbb{R}^{S \times d_{\text{model}}}$</span>, where <span class="math-container">$S$</span> is the number of tokens, and <span class="math-container">$d_{\text{model}}$</span> is the size of each token. Assuming you are referring to the PE defined in &quot;Attention is all you need&quot; (<a href="https://arxiv.org/pdf/1706.03762" rel="nofollow noreferrer">Vaswani et al., 2017</a>), then you have:
  <span class="math-container">$$\operatorname{PE}(p,2i) = \sin\left( \frac{p}{10,000^{\frac{2i}{d_{\text{model}}}}}\right)$$</span>
  <span class="math-container">$$\operatorname{PE}(p,2i+1) = \cos\left( \frac{p}{10,000^{\frac{2i}{d_{\text{model}}}}}\right)$$</span>
  where <span class="math-container">$d_{\text{model}}$</span> is the size of the embedding vector, <span class="math-container">$p$</span> is the position of the (token) embedding in the sequence, and <span class="math-container">$i$</span> is the feature index in the embedding space. As you can see, <span class="math-container">$\operatorname{PE}$</span> <strong>is not a function of the embeddings</strong>, but rather a determinsitc function of the position <span class="math-container">$p$</span> and the feature index <span class="math-container">$i$</span>. This means that <strong>PE is the same across all input sequences</strong>.</p>
  <p>This is also the reason why in many practical implementation, like <a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" rel="nofollow noreferrer">here</a>, PE is computed only once. Specifically, the PEs are precomputed for a fixed maximum sequence length and stored in a tensor. When an input sequence <span class="math-container">$x$</span> is processed, the corresponding PEs are sliced from this precomputed tensor (to match the dimensions) and added to <span class="math-container">$x$</span>.</p>
  <p>Therefore, back to your example, if your sequence is
  <span class="math-container">$$
  \begin{bmatrix}
  A \\
  B\\
  \end{bmatrix}
  =
  \begin{bmatrix}
  0.5 &amp; 0.0 \\
  1.0 &amp; 0.0 \\
  \end{bmatrix}
  $$</span>
  you have <span class="math-container">$d_{\text{model}} = 2$</span> and <span class="math-container">$p, i \in \{0, 1\}$</span>. First, compute the PEs:
  <span class="math-container">\begin{align}
  \operatorname{PE}(0,0) &amp;= \sin\left( \frac{0}{10,000^{\frac{0}{2}}} \right) = \sin\left( 0 \right) = 0 \\
  \operatorname{PE}(0,1) &amp;= \cos\left( \frac{0}{10,000^{\frac{0}{2}}} \right) = \cos\left( 0 \right) = 1 \\
  \operatorname{PE}(1,0) &amp;= \sin\left( \frac{1}{10,000^{\frac{0}{2}}} \right) = \sin(1) \approx 0.8415 \\
  \operatorname{PE}(1,1) &amp;= \cos\left( \frac{1}{10,000^{\frac{0}{2}}} \right) = \cos(1) \approx 0.5403
  \end{align}</span></p>
  <p>This results in the PE tensor:
  <span class="math-container">$$ 
  \operatorname{PE} = 
  \begin{bmatrix}
  0.0000 &amp; 1.0000\\
  0.8415 &amp; 0.5403\\
  \end{bmatrix}
  $$</span>
  which will be the same for all input sequences. Your input sequence is then transformed as follows:
  <span class="math-container">$$
  x' = x + \operatorname{PE} =
  \begin{bmatrix}
  0.5 &amp; 0.0 \\
  1.0 &amp; 0.0 \\
  \end{bmatrix}
  +
  \begin{bmatrix}
  0.0000 &amp; 1.0000\\
  0.8415 &amp; 0.5403\\
  \end{bmatrix}
  =
  \begin{bmatrix}
  0.5000 &amp; 1.0000\\
  1.8415 &amp; 0.5403\\
  \end{bmatrix}
  $$</span></p>
  <p>Notably, each PE is <strong>unique</strong> for each position within an input sequence. Consequently, if two distinct tokens within the same input sequence, denoted as <span class="math-container">$x_p$</span> and <span class="math-container">$x_q$</span>, yield identical embeddings after the application of their respective PEs:
  <span class="math-container">$$ x_p + \operatorname{PE}(p) = x_q + \operatorname{PE}(q)$$</span>
  several implications arise. First, the occurrence of such an event is highly improbable, given that <span class="math-container">$\operatorname{PE}(p) \neq \operatorname{PE}(q)$</span>. Second, even if this rare event occurs, the model is capable of distinguishing between the tokens based on their positional contexts. Since <span class="math-container">$\operatorname{PE}(p) \neq \operatorname{PE}(q)$</span>, then also <span class="math-container">$x_p \neq x_q$</span> (the embeddings are different). Thus, the information encoded by <span class="math-container">$x_p$</span> and <span class="math-container">$x_q$</span> are different, even though the final embeddings are the same.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-10T15:59:01.257Z
Answers.Text: What is meant by "$p$ is the position of the (token) embedding in the sequence"? "Position in the sequence" in what sense? The paper itself seems to be similarly vague about this.
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-10T16:04:25.473Z
Answers.Text: Given the input sequence $[a, b, c, d]$, $p=0$ for $a$, $p=1$ for $b$, $p=2$ for $c$, $p=3$ for $d$. Here, $a$, $b$, $c$, $d$ are your tokens, hence each is a vector of size $d_{\text{model}}$. Transformers expect a sequence of tokens as input, not one token at a time.
Answers.UserDisplayName: frad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-10T18:20:15.587Z
Answers.Text: If the final embeddings are the same, then how does the token embeddings $x_p$ and $x_q$ being different matter? Doesn’t the model use the final embedding state, not the intermediate token embedding states? Aren’t these intermediate token embedding states just a means to an end (the final embedding states)?
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-10T22:29:44.017Z
Answers.Text: Given $x_1 + y_1 = x_2 + y_2$ and $y_1 \neq y_2$, it follows that $x_1 \neq x_2$. If $x_1=x_2$, the initial equation results in $y_1=y_2$, which contradicts the initial assumption $y_1 \neq y_2$.
Answers.UserDisplayName: frad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-11T03:43:39.747Z
Answers.Text: The point that I'm making is that, if the (final) embeddings that the model uses are the same, then, even if the information encoded by $x_p$ and $x_q$ are different, I don't see how this allows the model to distinguish between the final embeddings (the embedding attained after element-wise addition of the token embedding and position encoding), since this intermediate state (the token embedding) is not the state that the model uses, since the model uses [...]
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-11T05:23:32.013Z
Answers.Text: [...] the final embeddings, which are attained after the element-wise addition, so the information that was available in that intermediate (token) embedding state is lost to the model when it uses the final embeddings. It seems to me that this leads to the original problem: identical (final) embeddings, despite different tokens/positions, with no way for the model to differentiate between them.
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-11T09:06:09.023Z
Answers.Text: I am not sure I understand your point. If positional information is embedded within the initial embeddings, then the intermediate embeddings will also inherently contain that information, as they are derived from these initial embeddings. This is a mathematical matter: you have two vectors, $x_p$ and $x_q$ occurring in two different positions, $p$ and $q$. This implies $p \neq q \Rightarrow \operatorname{PE}(p) \neq \operatorname{PE}(q)$ by definition of PE. [...]
Answers.UserDisplayName: frad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-11T09:06:19.297Z
Answers.Text: [...] By integrating the positional information into the token embeddings, we obtain $x_p + \operatorname{PE}(p)$ for the token at position $p$ and $x_q + \operatorname{PE}(q)$ for the token at position $q$. Given $\operatorname{PE}(p) \neq \operatorname{PE}(q)$ by definition, the equality $x_p + \operatorname{PE}(p) = x_q + \operatorname{PE}(q)$ holds iff $x_p \neq x_q$. This proves that in order to obtain the same representation for two tokens in two different positions, they must be different, thus they encode different information.
Answers.UserDisplayName: frad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-10T15:18:45.260Z
Answers.LastActivityDate: 2024-07-10T16:05:57.377Z
Answers.LastEditDate: 2024-07-10T16:05:57.377Z
Answers.UserDisplayName: frad
Answers.Body: <p>Long story short, they don't</p>
  <p>But probabilistically speaking, given that they are supposed to be in <span class="math-container">$\mathbb{R}^n$</span>, there is a probability 0 that they actually coincide</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-10T14:38:14.277Z
Answers.LastActivityDate: 2024-07-10T14:38:14.277Z
Answers.UserDisplayName: Alberto
Body: <p>In Transformer models, token embeddings are combined with positional encodings through element-wise addition to incorporate positional information. However, this raises a concern about the potential for different tokens in different positions to end up with identical embeddings.</p>
  <p>For example, consider the following:</p>
  <ul>
  <li>Embedding of token <span class="math-container">$A$</span>: <span class="math-container">$[0.5, 0]$</span></li>
  <li>Positional encoding at position <span class="math-container">$1$</span>: <span class="math-container">$[1, 0]$</span></li>
  <li>Combined representation for token <span class="math-container">$A$</span> at position <span class="math-container">$1$</span>: <span class="math-container">$[0.5 + 1, 0 + 0] = [1.5, 0]$</span></li>
  <li>Embedding of token <span class="math-container">$B$</span>: <span class="math-container">$[1, 0]$</span></li>
  <li>Positional encoding at position <span class="math-container">$2$</span>: <span class="math-container">$[0.5, 0]$</span></li>
  <li>Combined representation for token <span class="math-container">$B$</span> at position <span class="math-container">$2$</span>: <span class="math-container">$[1 + 0.5, 0 + 0] = [1.5, 0]$</span></li>
  </ul>
  <p>In this case, the combined representations for different tokens at different positions are the same.</p>
  <p>Given that positional encodings are added element-wise to token embeddings, how do Transformer models ensure the uniqueness of token representations to prevent different token-position combinations from resulting in the same embedding vector? Specifically, how do they handle cases where different embeddings and positional encodings could potentially sum to the same vector?</p>
  
Tags: <transformer><embeddings><positional-encoding>
Title: How do Transformer models ensure unique token representations when combining embeddings and positional encodings?
UserDisplayName: The Pointer
ThreadText: # How do Transformer models ensure unique token representations when combining embeddings and positional encodings?
  
  (Asked by: username_0 on 2024-07-10)
  
  In Transformer models, token embeddings are combined with positional encodings through element-wise addition to incorporate positional information. However, this raises a concern about the potential for different tokens in different positions to end up with identical embeddings.
  
  For example, consider the following:
  
  * Embedding of token $A$: $[0.5, 0]$
  * Positional encoding at position $1$: $[1, 0]$
  * Combined representation for token $A$ at position $1$: $[0.5 + 1, 0 + 0] = [1.5, 0]$
  * Embedding of token $B$: $[1, 0]$
  * Positional encoding at position $2$: $[0.5, 0]$
  * Combined representation for token $B$ at position $2$: $[1 + 0.5, 0 + 0] = [1.5, 0]$
  
  In this case, the combined representations for different tokens at different positions are the same.
  
  Given that positional encodings are added element-wise to token embeddings, how do Transformer models ensure the uniqueness of token representations to prevent different token-position combinations from resulting in the same embedding vector? Specifically, how do they handle cases where different embeddings and positional encodings could potentially sum to the same vector?
  
  **Tags:** transformer, embeddings, positional-encoding
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-07-10. Score: 0 | Accepted answer
  
  Positional Encoding (PE) is a broad term encompassing various techniques meant to embed positional information within a matrix $x \in \mathbb{R}^{S \times d\_{\text{model}}}$, where $S$ is the number of tokens, and $d\_{\text{model}}$ is the size of each token. Assuming you are referring to the PE defined in "Attention is all you need" ([Vaswani et al., 2017](https://arxiv.org/pdf/1706.03762)), then you have:
  $$\operatorname{PE}(p,2i) = \sin\left( \frac{p}{10,000^{\frac{2i}{d\_{\text{model}}}}}\right)$$
  $$\operatorname{PE}(p,2i+1) = \cos\left( \frac{p}{10,000^{\frac{2i}{d\_{\text{model}}}}}\right)$$
  where $d\_{\text{model}}$ is the size of the embedding vector, $p$ is the position of the (token) embedding in the sequence, and $i$ is the feature index in the embedding space. As you can see, $\operatorname{PE}$ **is not a function of the embeddings**, but rather a determinsitc function of the position $p$ and the feature index $i$. This means that **PE is the same across all input sequences**.
  
  This is also the reason why in many practical implementation, like [here](https://pytorch.org/tutorials/beginner/translation_transformer.html), PE is computed only once. Specifically, the PEs are precomputed for a fixed maximum sequence length and stored in a tensor. When an input sequence $x$ is processed, the corresponding PEs are sliced from this precomputed tensor (to match the dimensions) and added to $x$.
  
  Therefore, back to your example, if your sequence is
  $$
  \begin{bmatrix}
  A \\
  B\\
  \end{bmatrix}
  =
  \begin{bmatrix}
  0.5 & 0.0 \\
  1.0 & 0.0 \\
  \end{bmatrix}
  $$
  you have $d\_{\text{model}} = 2$ and $p, i \in \{0, 1\}$. First, compute the PEs:
  \begin{align}
  \operatorname{PE}(0,0) &= \sin\left( \frac{0}{10,000^{\frac{0}{2}}} \right) = \sin\left( 0 \right) = 0 \\
  \operatorname{PE}(0,1) &= \cos\left( \frac{0}{10,000^{\frac{0}{2}}} \right) = \cos\left( 0 \right) = 1 \\
  \operatorname{PE}(1,0) &= \sin\left( \frac{1}{10,000^{\frac{0}{2}}} \right) = \sin(1) \approx 0.8415 \\
  \operatorname{PE}(1,1) &= \cos\left( \frac{1}{10,000^{\frac{0}{2}}} \right) = \cos(1) \approx 0.5403
  \end{align}
  
  This results in the PE tensor:
  $$
  \operatorname{PE} =
  \begin{bmatrix}
  0.0000 & 1.0000\\
  0.8415 & 0.5403\\
  \end{bmatrix}
  $$
  which will be the same for all input sequences. Your input sequence is then transformed as follows:
  $$
  x' = x + \operatorname{PE} =
  \begin{bmatrix}
  0.5 & 0.0 \\
  1.0 & 0.0 \\
  \end{bmatrix}
  +
  \begin{bmatrix}
  0.0000 & 1.0000\\
  0.8415 & 0.5403\\
  \end{bmatrix}
  =
  \begin{bmatrix}
  0.5000 & 1.0000\\
  1.8415 & 0.5403\\
  \end{bmatrix}
  $$
  
  Notably, each PE is **unique** for each position within an input sequence. Consequently, if two distinct tokens within the same input sequence, denoted as $x\_p$ and $x\_q$, yield identical embeddings after the application of their respective PEs:
  $$ x\_p + \operatorname{PE}(p) = x\_q + \operatorname{PE}(q)$$
  several implications arise. First, the occurrence of such an event is highly improbable, given that $\operatorname{PE}(p) \neq \operatorname{PE}(q)$. Second, even if this rare event occurs, the model is capable of distinguishing between the tokens based on their positional contexts. Since $\operatorname{PE}(p) \neq \operatorname{PE}(q)$, then also $x\_p \neq x\_q$ (the embeddings are different). Thus, the information encoded by $x\_p$ and $x\_q$ are different, even though the final embeddings are the same.
  
  > **Comments:**
  > **username_0**: What is meant by "$p$ is the position of the (token) embedding in the sequence"? "Position in the sequence" in what sense? The paper itself seems to be similarly vague about this.  
  > **username_1**: Given the input sequence $[a, b, c, d]$, $p=0$ for $a$, $p=1$ for $b$, $p=2$ for $c$, $p=3$ for $d$. Here, $a$, $b$, $c$, $d$ are your tokens, hence each is a vector of size $d_{\text{model}}$. Transformers expect a sequence of tokens as input, not one token at a time.  
  > **username_0**: If the final embeddings are the same, then how does the token embeddings $x_p$ and $x_q$ being different matter? Doesn’t the model use the final embedding state, not the intermediate token embedding states? Aren’t these intermediate token embedding states just a means to an end (the final embedding states)?  
  > **username_1**: Given $x_1 + y_1 = x_2 + y_2$ and $y_1 \neq y_2$, it follows that $x_1 \neq x_2$. If $x_1=x_2$, the initial equation results in $y_1=y_2$, which contradicts the initial assumption $y_1 \neq y_2$.  
  > **username_0**: The point that I'm making is that, if the (final) embeddings that the model uses are the same, then, even if the information encoded by $x_p$ and $x_q$ are different, I don't see how this allows the model to distinguish between the final embeddings (the embedding attained after element-wise addition of the token embedding and position encoding), since this intermediate state (the token embedding) is not the state that the model uses, since the model uses [...]  
  > **username_0**: [...] the final embeddings, which are attained after the element-wise addition, so the information that was available in that intermediate (token) embedding state is lost to the model when it uses the final embeddings. It seems to me that this leads to the original problem: identical (final) embeddings, despite different tokens/positions, with no way for the model to differentiate between them.  
  > **username_1**: I am not sure I understand your point. If positional information is embedded within the initial embeddings, then the intermediate embeddings will also inherently contain that information, as they are derived from these initial embeddings. This is a mathematical matter: you have two vectors, $x_p$ and $x_q$ occurring in two different positions, $p$ and $q$. This implies $p \neq q \Rightarrow \operatorname{PE}(p) \neq \operatorname{PE}(q)$ by definition of PE. [...]  
  > **username_1**: [...] By integrating the positional information into the token embeddings, we obtain $x_p + \operatorname{PE}(p)$ for the token at position $p$ and $x_q + \operatorname{PE}(q)$ for the token at position $q$. Given $\operatorname{PE}(p) \neq \operatorname{PE}(q)$ by definition, the equality $x_p + \operatorname{PE}(p) = x_q + \operatorname{PE}(q)$ holds iff $x_p \neq x_q$. This proves that in order to obtain the same representation for two tokens in two different positions, they must be different, thus they encode different information.  
  
  ---
  
  ### Answer by username_2 on 2024-07-10. Score: 0
  
  Long story short, they don't
  
  But probabilistically speaking, given that they are supposed to be in $\mathbb{R}^n$, there is a probability 0 that they actually coincide
  
  
  ---
  
  