Answers.Body: <p>As far as I understood, the difference is the following: original Transformers use a fixed type of encoding, based on sine/cosine functions.</p>
  <p>On the other hand, GPT produces two embedding vectors: one of the input tokens, as usual in language models, and another for token positions themselves.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-30T22:19:13.970Z
Answers.LastActivityDate: 2021-12-31T09:12:25.093Z
Answers.LastEditDate: 2021-12-31T09:12:25.093Z
Answers.UserDisplayName: Leevo
Answers.Body: <p>The purpose of introduction of positional encoding  is to insert a notion of location of a given token in the sequence. Without it, due to the permutation equivariance (symmetry under the token permutation) there will be no notion of relative order inside a sequence.</p>
  <p>Given a token at <span class="math-container">$\text{pos}$</span>-th position we would like to make the model understand, that this token is at particular position. See pretty nice blog here - <a href="https://kazemnejad.com/blog/transformer_architecture_positional_encoding/" rel="noreferrer">https://kazemnejad.com/blog/transformer_architecture_positional_encoding/</a>.</p>
  <p><strong>Fixed encoding</strong></p>
  <p>In the <a href="https://arxiv.org/abs/1706.03762" rel="noreferrer">original Transformer</a> one uses a fixed map from the token position <span class="math-container">$i$</span> to the embedding vector added to the original embedding:
  <span class="math-container">$$
  \begin{aligned}
  PE(\text{pos}, 2i) &amp;= \sin(\text{pos} / 10000^{2i / d_{\text{model}}}) \\
  PE(\text{pos}, 2i + 1) &amp;= \cos(\text{pos} / 10000^{2i / d_{\text{model}}})
  \end{aligned}
  $$</span></p>
  <p>Here <span class="math-container">$\text{pos}$</span> is an index of the token in sequence, and <span class="math-container">$2i, 2i+1$</span> correspond to the dimension inside the embedding.</p>
  <p><strong>Learned encoding</strong></p>
  <p>Another strategy is to make map for <span class="math-container">$\text{pos}$</span> to the embedding vector of dimension <span class="math-container">$d_{\text{model}}$</span> <em>learnable</em>. One initializes somehow for each position in the sequence vector of positional embedding for each position from <span class="math-container">$0$</span> to <span class="math-container">$\text{max_length}$</span> and during the training these vectors are updated by gradient descent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-30T16:15:54.603Z
Answers.LastActivityDate: 2022-01-30T16:15:54.603Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>I know the original Transformer and the GPT (1-3) use two slightly different <strong>positional encoding</strong> techniques.</p>
  <p>More specifically, in GPT they say positional encoding is <em>learned</em>. What does that mean? OpenAI's papers don't go into detail very much.</p>
  <p>How do they really differ, mathematically speaking?</p>
  
Tags: <comparison><transformer><gpt><positional-encoding>
Title: What is the difference between the positional encoding techniques of the Transformer and GPT?
UserDisplayName: Leevo
ThreadText: # What is the difference between the positional encoding techniques of the Transformer and GPT?
  
  (Asked by: username_0 on 2020-11-23)
  
  I know the original Transformer and the GPT (1-3) use two slightly different **positional encoding** techniques.
  
  More specifically, in GPT they say positional encoding is *learned*. What does that mean? OpenAI's papers don't go into detail very much.
  
  How do they really differ, mathematically speaking?
  
  **Tags:** comparison, transformer, gpt, positional-encoding
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-01-30. Score: 6 | Accepted answer
  
  The purpose of introduction of positional encoding is to insert a notion of location of a given token in the sequence. Without it, due to the permutation equivariance (symmetry under the token permutation) there will be no notion of relative order inside a sequence.
  
  Given a token at $\text{pos}$-th position we would like to make the model understand, that this token is at particular position. See pretty nice blog here - <https://kazemnejad.com/blog/transformer_architecture_positional_encoding/>.
  
  **Fixed encoding**
  
  In the [original Transformer](https://arxiv.org/abs/1706.03762) one uses a fixed map from the token position $i$ to the embedding vector added to the original embedding:
  $$
  \begin{aligned}
  PE(\text{pos}, 2i) &= \sin(\text{pos} / 10000^{2i / d\_{\text{model}}}) \\
  PE(\text{pos}, 2i + 1) &= \cos(\text{pos} / 10000^{2i / d\_{\text{model}}})
  \end{aligned}
  $$
  
  Here $\text{pos}$ is an index of the token in sequence, and $2i, 2i+1$ correspond to the dimension inside the embedding.
  
  **Learned encoding**
  
  Another strategy is to make map for $\text{pos}$ to the embedding vector of dimension $d\_{\text{model}}$ *learnable*. One initializes somehow for each position in the sequence vector of positional embedding for each position from $0$ to $\text{max\_length}$ and during the training these vectors are updated by gradient descent.
  
  
  ---
  
  ### Answer by username_0 on 2021-11-30. Score: 0
  
  As far as I understood, the difference is the following: original Transformers use a fixed type of encoding, based on sine/cosine functions.
  
  On the other hand, GPT produces two embedding vectors: one of the input tokens, as usual in language models, and another for token positions themselves.
  
  
  ---
  
  