Answers.Body: <p>In high-dimensional spaces, the token embeddings and positional encodings can be thought of as forming two separate subspaces that are approximately orthogonal to each other. This is based on the mathematical property that</p>
  <blockquote>
  <p>Randomly chosen vectors in high dimensions are almost approximately
  orthogonal <a href="https://mathoverflow.net/questions/248466/why-are-two-random-vectors-in-mathbb-rn-approximately-orthogonal-for-large">[1]</a>.</p>
  </blockquote>
  <p>This means that these subspaces can be manipulated essentially independently of each other by the same learned transformation, despite the addition operation.</p>
  <p>So, by adding the embeddings, we’re not losing any information. Instead, we’re allowing the model to decide how best to use the information from the token embeddings and the positional encodings. This is more efficient than concatenation, which would increase the dimensionality and the number of parameters the model needs to learn.</p>
  <p>In essence, the addition of positional encodings to token embeddings in Transformer models might seem counterintuitive, but it’s a clever trick that leverages the properties of high-dimensional spaces to efficiently combine different types of information.</p>
  <p>P.S. - There isn’t a definitive answer to this question. However, after extensive research, I found this intuitive discussion here <a href="https://www.reddit.com/r/MachineLearning/comments/cttefo/comment/exs7d08/?utm_source=reddit&amp;utm_medium=web2x&amp;context=3" rel="noreferrer">[2]</a> that beautifully conveys the idea explained above.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-24T15:30:11.497Z
Answers.LastActivityDate: 2023-12-24T15:30:11.497Z
Answers.UserDisplayName: Abdur Rahman
Answers.Body: <p>First of all, I think it is very hard to properly reason about these things, but there are a few points that might justify using sum instead of concatenation.
  For example, concatenation would have the drawback of increasing the dimensionality. So for subsequent residual connections to work, you would either have to use the increased dimensionality throughout the model, or add yet another layer to transform it back to the original dimensionality.</p>
  <blockquote>
  <p>The thing is, these embeddings carry different types of information, so intuitively adding them together doesn't really make sense. I mean, you cannot add 2 meters to 3 kilograms, but you can make a tuple</p>
  </blockquote>
  <p>I would say that because the token-embedding is learned, you cannot really compare it to a fix unit like kilogram. Instead the embedding space of the token can be optimized to work with the positional encoding under summation.</p>
  <blockquote>
  <p>By adding them together, we are assuming the information about token, segmentation, and position can be simultaneously represented in the same embedding space, but that sounds like a bold claim.</p>
  </blockquote>
  <p>The same applies here, the problem is not to embed them into the same space, but rather that subsequent layers can separate the position information from the token information.
  And I think this is possible for two reasons. Firstly, if you look at the visual representation of the positional embedding, the highest distortion by summation would happen in the first dimensions:</p>
  <p><img src="https://www.inovex.de/wp-content/uploads/Untitled.png" alt="positional_encoding" /> (Image taken from <a href="https://www.inovex.de/de/blog/positional-encoding-everything-you-need-to-know/" rel="nofollow noreferrer">here</a>)</p>
  <p>Therefore the token embedding could learn to encode high-frequency information only in the last dimensions to be less affected by the positional embedding.
  I think another interesting statement in the Transformer paper is that the positional encoding behaves linearly w.r.t. relative position:</p>
  <blockquote>
  <p>We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset <span class="math-container">$k$</span>, <span class="math-container">$PE_{pos+k}$</span> can be represented as a linear function of <span class="math-container">$PE_{pos}$</span>.
  <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">[Source: Transformer Paper]</a></p>
  </blockquote>
  <p>So this property shouldn't add additional non linearity to the token embedding, but instead acts more like a linear transformation, since any change in position changes the the embedding linearly. In my intuition this should also enable easy separation of positional vs token information.</p>
  <p><em>This is my intuition so far, I am happy to hear your thoughts and additions</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T09:04:56.977Z
Answers.LastActivityDate: 2022-06-21T09:04:56.977Z
Answers.UserDisplayName: Chillston
Answers.Body: <p>The confusion here is that we believe positional embedding is a more complicated version of adding positional information to the work embedding; however, it is not actually. Adding new dimensions to each embedding increases the dimensionality of the problem. On the other hand, please note that the added positional embedding is static, as shown in this image for a 2D positional embedding:</p>
  <p><a href="https://i.sstatic.net/zbcvp.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zbcvp.png" alt="enter image description here" /></a></p>
  <p>The added positional embeddings are the same for all the inputs, and the transformer can separate the positional information from the actual work embedding through the training process. Therefore, the positional embedding doesn't mess with the work embedding information, and adding them is a more efficient way of adding the positional information that concatenates them.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-08T12:22:23.033Z
Answers.Text: Could you elaborate how does the training process can learn to separate the positional and word embedding information?
Answers.UserDisplayName: Glue
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-09T23:58:48.557Z
Answers.Text: @Glue The positional embedding is like a bias added to all word embeddings located at each specific position. It is like adding a constant to your inputs. Not sure if this actually happens or not, but separating them is like learning a bias. The value of bias is your positional embedding, and what remains after removing the bias is your word embedding.
Answers.UserDisplayName: Hamid Mohammadi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-17T15:14:33.063Z
Answers.LastActivityDate: 2022-12-17T15:14:33.063Z
Answers.UserDisplayName: Hamid Mohammadi
Body: <p>Let's consider the following example from BERT</p>
  <p><a href="https://i.sstatic.net/YLGSz.png" rel="noreferrer"><img src="https://i.sstatic.net/YLGSz.png" alt="enter image description here" /></a></p>
  <p>I cannot understand why &quot;the input embeddings are the <em>sum</em> of the token embeddings, the segmentation embeddings, and the position embeddings&quot;. The thing is, these embeddings carry different types of information, so intuitively adding them together doesn't really make sense. I mean, you cannot add 2 meters to 3 kilograms, but you can make a tuple (2 meters, 3 kilograms), so I think it's more natural to concatenate these embedding together. By adding them together, we are assuming the information about token, segmentation, and position can be simultaneously represented in the same embedding space, but that sounds like a bold claim.</p>
  <p>Other transformers, like ViTMAE, seem to follow the trend of <a href="https://github.com/huggingface/transformers/blob/v4.20.0/src/transformers/models/vit_mae/modeling_vit_mae.py#L795" rel="noreferrer">adding position embeddings to other &quot;semantic&quot; embeddings</a>. What's the rationale behind the practice?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-20T06:04:16.267Z
Comments.Text: Can I know why you are comparing with physical units?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-20T06:18:09.400Z
Comments.Text: @hanugm Sorry for the confusion! I was analogizing 2 meters to token embedding, and 3 kilograms to position embedding. The idea is that token/position embeddings respectively describe the semantics/position of the tokens, so they should have different units, just like length and weight have meter and kilogram as their units. In [dimensional analysis](https://en.wikipedia.org/wiki/Dimensional_analysis), it is established that you cannot add quantities with different units together (i.e. dimensional homogeneity), which is why I think it doesn't make sense to add embeddings together.
Comments.UserDisplayName: nalzok
Tags: <neural-networks><deep-learning><transformer><attention><embeddings>
Title: Why are embeddings added, not concatenated?
UserDisplayName: nalzok
ThreadText: # Why are embeddings added, not concatenated?
  
  (Asked by: username_0 on 2022-06-20)
  
  Let's consider the following example from BERT
  
  [![enter image description here](https://i.sstatic.net/YLGSz.png)](https://i.sstatic.net/YLGSz.png)
  
  I cannot understand why "the input embeddings are the *sum* of the token embeddings, the segmentation embeddings, and the position embeddings". The thing is, these embeddings carry different types of information, so intuitively adding them together doesn't really make sense. I mean, you cannot add 2 meters to 3 kilograms, but you can make a tuple (2 meters, 3 kilograms), so I think it's more natural to concatenate these embedding together. By adding them together, we are assuming the information about token, segmentation, and position can be simultaneously represented in the same embedding space, but that sounds like a bold claim.
  
  Other transformers, like ViTMAE, seem to follow the trend of [adding position embeddings to other "semantic" embeddings](https://github.com/huggingface/transformers/blob/v4.20.0/src/transformers/models/vit_mae/modeling_vit_mae.py#L795). What's the rationale behind the practice?
  
  **Tags:** neural-networks, deep-learning, transformer, attention, embeddings
  > **Comments:**
  > **username_1**: Can I know why you are comparing with physical units?  
  > **username_0**: @hanugm Sorry for the confusion! I was analogizing 2 meters to token embedding, and 3 kilograms to position embedding. The idea is that token/position embeddings respectively describe the semantics/position of the tokens, so they should have different units, just like length and weight have meter and kilogram as their units. In [dimensional analysis](https://en.wikipedia.org/wiki/Dimensional_analysis), it is established that you cannot add quantities with different units together (i.e. dimensional homogeneity), which is why I think it doesn't make sense to add embeddings together.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2023-12-24. Score: 8
  
  In high-dimensional spaces, the token embeddings and positional encodings can be thought of as forming two separate subspaces that are approximately orthogonal to each other. This is based on the mathematical property that
  
  > Randomly chosen vectors in high dimensions are almost approximately
  > orthogonal [[1]](https://mathoverflow.net/questions/248466/why-are-two-random-vectors-in-mathbb-rn-approximately-orthogonal-for-large).
  
  This means that these subspaces can be manipulated essentially independently of each other by the same learned transformation, despite the addition operation.
  
  So, by adding the embeddings, we’re not losing any information. Instead, we’re allowing the model to decide how best to use the information from the token embeddings and the positional encodings. This is more efficient than concatenation, which would increase the dimensionality and the number of parameters the model needs to learn.
  
  In essence, the addition of positional encodings to token embeddings in Transformer models might seem counterintuitive, but it’s a clever trick that leverages the properties of high-dimensional spaces to efficiently combine different types of information.
  
  P.S. - There isn’t a definitive answer to this question. However, after extensive research, I found this intuitive discussion here [[2]](https://www.reddit.com/r/MachineLearning/comments/cttefo/comment/exs7d08/?utm_source=reddit&utm_medium=web2x&context=3) that beautifully conveys the idea explained above.
  
  
  ---
  
  ### Answer by username_3 on 2022-06-21. Score: 2
  
  First of all, I think it is very hard to properly reason about these things, but there are a few points that might justify using sum instead of concatenation.
  For example, concatenation would have the drawback of increasing the dimensionality. So for subsequent residual connections to work, you would either have to use the increased dimensionality throughout the model, or add yet another layer to transform it back to the original dimensionality.
  
  > The thing is, these embeddings carry different types of information, so intuitively adding them together doesn't really make sense. I mean, you cannot add 2 meters to 3 kilograms, but you can make a tuple
  
  I would say that because the token-embedding is learned, you cannot really compare it to a fix unit like kilogram. Instead the embedding space of the token can be optimized to work with the positional encoding under summation.
  
  > By adding them together, we are assuming the information about token, segmentation, and position can be simultaneously represented in the same embedding space, but that sounds like a bold claim.
  
  The same applies here, the problem is not to embed them into the same space, but rather that subsequent layers can separate the position information from the token information.
  And I think this is possible for two reasons. Firstly, if you look at the visual representation of the positional embedding, the highest distortion by summation would happen in the first dimensions:
  
  ![positional_encoding](https://www.inovex.de/wp-content/uploads/Untitled.png) (Image taken from [here](https://www.inovex.de/de/blog/positional-encoding-everything-you-need-to-know/))
  
  Therefore the token embedding could learn to encode high-frequency information only in the last dimensions to be less affected by the positional embedding.
  I think another interesting statement in the Transformer paper is that the positional encoding behaves linearly w.r.t. relative position:
  
  > We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset $k$, $PE\_{pos+k}$ can be represented as a linear function of $PE\_{pos}$.
  > [[Source: Transformer Paper]](https://arxiv.org/abs/1706.03762)
  
  So this property shouldn't add additional non linearity to the token embedding, but instead acts more like a linear transformation, since any change in position changes the the embedding linearly. In my intuition this should also enable easy separation of positional vs token information.
  
  *This is my intuition so far, I am happy to hear your thoughts and additions*
  
  
  ---
  
  ### Answer by username_4 on 2022-12-17. Score: 0
  
  The confusion here is that we believe positional embedding is a more complicated version of adding positional information to the work embedding; however, it is not actually. Adding new dimensions to each embedding increases the dimensionality of the problem. On the other hand, please note that the added positional embedding is static, as shown in this image for a 2D positional embedding:
  
  [![enter image description here](https://i.sstatic.net/zbcvp.png)](https://i.sstatic.net/zbcvp.png)
  
  The added positional embeddings are the same for all the inputs, and the transformer can separate the positional information from the actual work embedding through the training process. Therefore, the positional embedding doesn't mess with the work embedding information, and adding them is a more efficient way of adding the positional information that concatenates them.
  
  > **Comments:**
  > **username_5**: Could you elaborate how does the training process can learn to separate the positional and word embedding information?  
  > **username_4**: @Glue The positional embedding is like a bias added to all word embeddings located at each specific position. It is like adding a constant to your inputs. Not sure if this actually happens or not, but separating them is like learning a bias. The value of bias is your positional embedding, and what remains after removing the bias is your word embedding.  
  
  ---
  
  