Answers.Body: <p>Depends... for ViT yes, they are called learned positional encodings, and the reason is that you always have a fixed number of patches (at least theoretically)</p>
  <p>However, for NLP, you cannot do that, because at inference, you might have strings longer or shorter than the reference, and if you have trained only <code>seq_len</code> positional encoding, and the inference string is longer than that, than you won't be able to apply a positional encoding to the final part of it</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T08:40:34.787Z
Answers.LastActivityDate: 2024-01-26T08:40:34.787Z
Answers.UserDisplayName: Alberto
Body: <p>Here's a basic GPT2 implementation:</p>
  <pre><code>class GPT(nn.Module):
      def __init__(self, vocab_size, seq_len, model_dim, n_heads, n_layers):
          super().__init__()
          self.seq_len = seq_len
          self.wte = nn.Embedding(vocab_size, model_dim)
          self.wpe = nn.Embedding(seq_len, model_dim)
          self.h = nn.Sequential(
              *[DecodeLayer(model_dim, n_heads) for _ in range(n_layers)]
          )
          self.ln_f = nn.LayerNorm(model_dim)
  
      def forward(self, x):
          embeds = self.wte(x) + self.wpe(torch.arange(0, self.seq_len).to(x.device)) # [seq_len, model_dim]
          embeds = self.h(embeds)
          embeds = self.ln_f(embeds)
          # (seq_len, model_dim)  x (model_dim, vocab_size) =&gt; seq_len vocab_size
          return embeds @ self.wte.weight.T
  </code></pre>
  <p>The positional embeddings are added to the input with:</p>
  <pre><code>self.wpe(torch.arange(0, self.seq_len).to(x.device))
  </code></pre>
  <p>I wonder, if it would be possible to just do: <code>self.wpe = nn.Parameter(seq_len, model_dim)</code>, and change the line in the forward pass to:</p>
  <pre><code>embeds = self.wte(x) + self.wpe
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-25T23:32:59.850Z
Comments.Text: Is this an implementation question or a theory question? i.e., are you asking whether the two implementations do the same thing or are you asking whether a slightly different way of doing PE would work effectively.
Comments.UserDisplayName: Alexander Wan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-26T05:11:12.573Z
Comments.Text: Good question. There's 2 questions hidden in here:
  
  Implementation question -- the 2 things should do the same thing as far as I can tell. Since, `torch.arange(0, self.seq_len)` will select the entire embedding matrix. However, from some testing, they do not do the same thing.
  
  Theory question -- why not just add a `seq_len, model_dim` size matrix to the output of `self.wte(x)`, will it not have the same effect as using an embedding layer?
Comments.UserDisplayName: Foobar
Tags: <transformer><backpropagation>
Title: Can positional encodings in transformers be added
UserDisplayName: Foobar
ThreadText: # Can positional encodings in transformers be added
  
  (Asked by: username_0 on 2024-01-25)
  
  Here's a basic GPT2 implementation:
  class GPT(nn.Module):
      def __init__(self, vocab_size, seq_len, model_dim, n_heads, n_layers):
          super().__init__()
          self.seq_len = seq_len
          self.wte = nn.Embedding(vocab_size, model_dim)
          self.wpe = nn.Embedding(seq_len, model_dim)
          self.h = nn.Sequential(
              *[DecodeLayer(model_dim, n_heads) for _ in range(n_layers)]
          )
          self.ln_f = nn.LayerNorm(model_dim)
  
      def forward(self, x):
          embeds = self.wte(x) + self.wpe(torch.arange(0, self.seq_len).to(x.device)) # [seq_len, model_dim]
          embeds = self.h(embeds)
          embeds = self.ln_f(embeds)
          # (seq_len, model_dim)  x (model_dim, vocab_size) => seq_len vocab_size
          return embeds @ self.wte.weight.T
  
  The positional embeddings are added to the input with:
  self.wpe(torch.arange(0, self.seq_len).to(x.device))
  
  I wonder, if it would be possible to just do: self.wpe = nn.Parameter(seq_len, model_dim), and change the line in the forward pass to:
  embeds = self.wte(x) + self.wpe
  
  **Tags:** transformer, backpropagation
  > **Comments:**
  > **username_1**: Is this an implementation question or a theory question? i.e., are you asking whether the two implementations do the same thing or are you asking whether a slightly different way of doing PE would work effectively.  
  > **username_0**: Good question. There's 2 questions hidden in here:  Implementation question -- the 2 things should do the same thing as far as I can tell. Since, `torch.arange(0, self.seq_len)` will select the entire embedding matrix. However, from some testing, they do not do the same thing.  Theory question -- why not just add a `seq_len, model_dim` size matrix to the output of `self.wte(x)`, will it not have the same effect as using an embedding layer?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-01-26. Score: 1
  
  Depends... for ViT yes, they are called learned positional encodings, and the reason is that you always have a fixed number of patches (at least theoretically)
  
  However, for NLP, you cannot do that, because at inference, you might have strings longer or shorter than the reference, and if you have trained only `seq_len` positional encoding, and the inference string is longer than that, than you won't be able to apply a positional encoding to the final part of it
  
  
  ---
  
  