Answers.Body: <p>transformers models use the sinusoidal position encoding to capture the position of each token, and this method has some limits :<br></p>
  <ol>
  <li>the sinusoidal position encoding is manually designed so it is not flexible enough and does not contain any learnable parameters</li>
  <li>if we have a long sequence so we may have the same vector position because sin and cos are periodic function, so we need unique position vector for each token and the perfect way to do that is using learnable position encoding</li>
  <li>and this position encoding is independent on dataset, sometimes if we change the position of one token in sentence there is no different in meaning but sometimes the meaning of sentence change, so it depend on dataset, for that the  position encoding should be learnable from data<br>
  so this is the main principle issue that we have with the traditional method of position encoding and for that many of researcher try to introduce some solution and use the learnable position encoding, this paper may help  better understand the logic behind this idea <a href="http://proceedings.mlr.press/v119/liu20n/liu20n.pdf" rel="nofollow noreferrer">http://proceedings.mlr.press/v119/liu20n/liu20n.pdf</a></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-13T21:09:18.983Z
Answers.LastActivityDate: 2024-04-13T21:09:18.983Z
Answers.UserDisplayName: LAILA EL OUEDEGHYRY
Body: <p>In the original paper of <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">transformers</a> they using positional encoding to capture the position of each word in the sentence and for calculate that it using sin and cos ,like shom in the image.<a href="https://i.sstatic.net/pFtun.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pFtun.png" alt="Sinusoidal positional encoding" /></a> In Bert and the author architecture that based on transformers ,they use learnable position encoding ,so they initialize the vectors of positional encoding randomly and then start to adjust them in training.<br>My question is why we use the learnable positional encoding, what is the objective?</p>
  
Tags: <machine-learning><deep-learning><natural-language-processing><transformer><bert>
Title: why we use learnable positional encoding instead of Sinusoidal positional encoding
UserDisplayName: LAILA EL OUEDEGHYRY
ThreadText: # why we use learnable positional encoding instead of Sinusoidal positional encoding
  
  (Asked by: username_0 on 2024-04-09)
  
  In the original paper of [transformers](https://arxiv.org/pdf/1706.03762.pdf) they using positional encoding to capture the position of each word in the sentence and for calculate that it using sin and cos ,like shom in the image.[![Sinusoidal positional encoding](https://i.sstatic.net/pFtun.png)](https://i.sstatic.net/pFtun.png) In Bert and the author architecture that based on transformers ,they use learnable position encoding ,so they initialize the vectors of positional encoding randomly and then start to adjust them in training.  
  My question is why we use the learnable positional encoding, what is the objective?
  
  **Tags:** machine-learning, deep-learning, natural-language-processing, transformer, bert
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-04-13. Score: 1
  
  transformers models use the sinusoidal position encoding to capture the position of each token, and this method has some limits :
  
  1. the sinusoidal position encoding is manually designed so it is not flexible enough and does not contain any learnable parameters
  2. if we have a long sequence so we may have the same vector position because sin and cos are periodic function, so we need unique position vector for each token and the perfect way to do that is using learnable position encoding
  3. and this position encoding is independent on dataset, sometimes if we change the position of one token in sentence there is no different in meaning but sometimes the meaning of sentence change, so it depend on dataset, for that the position encoding should be learnable from data  
     so this is the main principle issue that we have with the traditional method of position encoding and for that many of researcher try to introduce some solution and use the learnable position encoding, this paper may help better understand the logic behind this idea <http://proceedings.mlr.press/v119/liu20n/liu20n.pdf>
  
  
  ---
  
  