Answers.Body: <p>Absolute position embeddings capture the absolute location of a token. Absolute location would refer to e.g., the 1st, 2nd, 3rd token etc. The sinusoidal embeddings in Vaswani's paper capture this absolute position information. But, if you have absolute position encodings, then you can also always derive relative positions, and sinusoidal embeddings make that really easy, but because absolute position is encoded, it would be considered an absolute position encoding.</p>
  <p>Contrast that with <a href="https://paperswithcode.com/method/relative-position-encodings" rel="nofollow noreferrer">relative position encoding</a>, where <em>only</em> the relative position between two tokens is used. e.g., <a href="https://arxiv.org/pdf/1803.02155.pdf" rel="nofollow noreferrer">in the paper</a>, the embeddings are only used during the attention operation, and they only capture information about the distance between two tokens.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-19T19:35:19.897Z
Answers.LastActivityDate: 2024-01-19T19:35:19.897Z
Answers.UserDisplayName: Alexander Wan
Body: <p>I am currently reading in depth about positional encodings, and as we know there are two types of positional encodings: Absolute and relative.</p>
  <h4>My question:</h4>
  <p>Why is the sinusoidal model classified as absolute positional encoding in some literature, given that in Vaswani's original paper it was said that it captures relative relationships between words, and this has been proven <a href="https://blog.timodenk.com/linear-relationships-in-the-transformers-positional-encoding/" rel="nofollow noreferrer">here</a>.</p>
  <p>However, while I was reading a <a href="https://direct.mit.edu/coli/article/48/3/733/111478/Position-Information-in-Transformers-An-Overview" rel="nofollow noreferrer">research</a>, it was mentioned that projections that occur in the attention layer destroy this:</p>
  <blockquote>
  <p>Indeed, sinusoidal position embeddings exhibit useful properties in theory. Yan et al. (2019) investigate the dot product of sinusoidal position embeddings and prove important properties:
  (1) The dot product of two sinusoidal position embeddings depends only on their relative distance. That is, <img src="https://latex.codecogs.com/svg.image?&amp;space;P_t%5ET.P_%7Bt+r%7D" alt=" P_t^T.P_{t+r}" /> is independent of <img src="https://latex.codecogs.com/svg.image?t" alt="t" />.
  (2) <img src="https://latex.codecogs.com/svg.image?&amp;space;P_t%5ET.P_%7Bt+r%7D=P_t%5ET.P_%7Bt-r%7D" alt="P_t^T.P_{t+r}=P_t^T.P_{t-r}" />, which means that sinusoidal position embeddings are unaware of direction. However, in practice the sinusoidal embeddings are projected with two different projection matrices, which destroys these properties.</p>
  </blockquote>
  <p>Is this the reason?</p>
  
Tags: <natural-language-processing><transformer><positional-encoding>
Title: Why is the sinusoidal model classified as absolute positional encoding in some literature?
UserDisplayName: Ali Haider Ahmad
ThreadText: # Why is the sinusoidal model classified as absolute positional encoding in some literature?
  
  (Asked by: username_0 on 2024-01-05)
  
  I am currently reading in depth about positional encodings, and as we know there are two types of positional encodings: Absolute and relative.
  
  #### My question:
  
  Why is the sinusoidal model classified as absolute positional encoding in some literature, given that in Vaswani's original paper it was said that it captures relative relationships between words, and this has been proven [here](https://blog.timodenk.com/linear-relationships-in-the-transformers-positional-encoding/).
  
  However, while I was reading a [research](https://direct.mit.edu/coli/article/48/3/733/111478/Position-Information-in-Transformers-An-Overview), it was mentioned that projections that occur in the attention layer destroy this:
  
  > Indeed, sinusoidal position embeddings exhibit useful properties in theory. Yan et al. (2019) investigate the dot product of sinusoidal position embeddings and prove important properties:
  > (1) The dot product of two sinusoidal position embeddings depends only on their relative distance. That is, ![ P_t^T.P_{t+r}](https://latex.codecogs.com/svg.image?&space;P_t%5ET.P_%7Bt+r%7D) is independent of ![t](https://latex.codecogs.com/svg.image?t).
  > (2) ![P_t^T.P_{t+r}=P_t^T.P_{t-r}](https://latex.codecogs.com/svg.image?&space;P_t%5ET.P_%7Bt+r%7D=P_t%5ET.P_%7Bt-r%7D), which means that sinusoidal position embeddings are unaware of direction. However, in practice the sinusoidal embeddings are projected with two different projection matrices, which destroys these properties.
  
  Is this the reason?
  
  **Tags:** natural-language-processing, transformer, positional-encoding
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-19. Score: 1 | Accepted answer
  
  Absolute position embeddings capture the absolute location of a token. Absolute location would refer to e.g., the 1st, 2nd, 3rd token etc. The sinusoidal embeddings in Vaswani's paper capture this absolute position information. But, if you have absolute position encodings, then you can also always derive relative positions, and sinusoidal embeddings make that really easy, but because absolute position is encoded, it would be considered an absolute position encoding.
  
  Contrast that with [relative position encoding](https://paperswithcode.com/method/relative-position-encodings), where *only* the relative position between two tokens is used. e.g., [in the paper](https://arxiv.org/pdf/1803.02155.pdf), the embeddings are only used during the attention operation, and they only capture information about the distance between two tokens.
  
  
  ---
  
  