Answers.Body: <p>The current code just implements the now-standard expressions for positional embeddings given in the original transformer paper (<a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">Attention is all you need</a>).</p>
  <p>In section 3.5 of this paper they suggest the formula
  <span class="math-container">$$PE_{(pos,2i)} = \sin \left(\frac{pos}{10000^{2i/d_{model}}}\right)$$</span>
  <span class="math-container">$$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$</span></p>
  <p>It turns out to be easier (or more efficient?) to implement in code using the standard maths identities <span class="math-container">$x = \exp \log x$</span> and <span class="math-container">$\log x^a = a \log x$</span>
  <span class="math-container">$$\frac{pos}{10000^{2i/d_{model}}} = \exp\left(\log(pos) - \frac{2i}{d_{model}} \log(10000) \right)$$</span></p>
  <p>Hopefully you can now see that this corresponds to the code you have given.</p>
  <p>But perhaps you are interested in the question as to why the positional encodings are of this fairly whacky form. My understanding of the main intuitions are that sines and cosines interact really nicely with translation due to their periodicity, and by using exponentially spaced 'frequencies' one can extract signals for interactions at a large number of different 'length scales'; see <a href="https://kazemnejad.com/blog/transformer_architecture_positional_encoding/" rel="nofollow noreferrer">this excellent blog post and associated links</a> for further explanation. But as I understand it, the main reason this clever idea is so widely used is because it works so well in practice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-07T10:39:21.510Z
Answers.LastActivityDate: 2023-08-07T10:39:21.510Z
Answers.UserDisplayName: just another mathmo
Body: <p>This code snippet is from <a href="https://huggingface.co/blog/annotated-diffusion" rel="nofollow noreferrer">here</a> under the section named &quot;Position embeddings&quot;.</p>
  <pre><code>class SinusoidalPositionEmbeddings(nn.Module):
      def __init__(self, dim):
          super().__init__()
          self.dim = dim
  
      def forward(self, time):
          device = time.device
          half_dim = self.dim // 2
          embeddings = math.log(10000) / (half_dim - 1)
          embeddings = torch.exp(torch.arange(half_dim, device=device) * -embeddings)
          embeddings = time[:, None] * embeddings[None, :]
          embeddings = torch.cat((embeddings.sin(), embeddings.cos()), dim=-1)
          return embeddings
  </code></pre>
  <p>The code is implementing Positional Encoding that was introduced in Transformer model.<br />
  I don't understand why we have to use <code>torch.exp</code> and <code>math.log</code>. Is this related to scaling or non-negative issue?</p>
  <p>Many thanks ahead!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-24T18:40:17.907Z
Comments.Text: This problem had been explained in the blog: https://medium.com/@hunter-j-phillips/positional-encoding-7a93db4109e6
Comments.UserDisplayName: 黄祖荣
Tags: <transformer><positional-encoding>
Title: Why use exponential and log in Positional Encoding of Transformer
UserDisplayName: Jun
ThreadText: # Why use exponential and log in Positional Encoding of Transformer
  
  (Asked by: username_0 on 2023-08-06)
  
  This code snippet is from here under the section named "Position embeddings".
  class SinusoidalPositionEmbeddings(nn.Module):
      def __init__(self, dim):
          super().__init__()
          self.dim = dim
  
      def forward(self, time):
          device = time.device
          half_dim = self.dim // 2
          embeddings = math.log(10000) / (half_dim - 1)
          embeddings = torch.exp(torch.arange(half_dim, device=device) * -embeddings)
          embeddings = time[:, None] * embeddings[None, :]
          embeddings = torch.cat((embeddings.sin(), embeddings.cos()), dim=-1)
          return embeddings
  
  The code is implementing Positional Encoding that was introduced in Transformer model.
  I don't understand why we have to use torch.exp and math.log. Is this related to scaling or non-negative issue?
  Many thanks ahead!
  
  **Tags:** transformer, positional-encoding
  > **Comments:**
  > **username_1**: This problem had been explained in the blog: https://medium.com/@hunter-j-phillips/positional-encoding-7a93db4109e6  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-08-07. Score: 1
  
  The current code just implements the now-standard expressions for positional embeddings given in the original transformer paper ([Attention is all you need](https://arxiv.org/pdf/1706.03762.pdf)).
  
  In section 3.5 of this paper they suggest the formula
  $$PE\_{(pos,2i)} = \sin \left(\frac{pos}{10000^{2i/d\_{model}}}\right)$$
  $$PE\_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d\_{model}}}\right)$$
  
  It turns out to be easier (or more efficient?) to implement in code using the standard maths identities $x = \exp \log x$ and $\log x^a = a \log x$
  $$\frac{pos}{10000^{2i/d\_{model}}} = \exp\left(\log(pos) - \frac{2i}{d\_{model}} \log(10000) \right)$$
  
  Hopefully you can now see that this corresponds to the code you have given.
  
  But perhaps you are interested in the question as to why the positional encodings are of this fairly whacky form. My understanding of the main intuitions are that sines and cosines interact really nicely with translation due to their periodicity, and by using exponentially spaced 'frequencies' one can extract signals for interactions at a large number of different 'length scales'; see [this excellent blog post and associated links](https://kazemnejad.com/blog/transformer_architecture_positional_encoding/) for further explanation. But as I understand it, the main reason this clever idea is so widely used is because it works so well in practice.
  
  
  ---
  
  