Body: <p>I'm a bit confused how the position embedding in happened to each patch in the transformer. I thought Ideally we'd want each patch to have a value of (1, 2, 3, 4....) to describe the position of the patch in the image. but from the implementation <a href="https://github.com/gupta-abhay/pytorch-vit/blob/main/vit/patch_embed.py" rel="nofollow noreferrer">here</a> there do something like this:</p>
  <pre><code> # positional embedding
              self.pos_embed = nn.Parameter(
                  torch.zeros(1, num_patches, embedding_dim)
              )
  </code></pre>
  <p>Which is quite confusing because now we have some sort of mapping instead of just a value appended to each patch. Also, there is some sort of implicit position appended to the patch right? Assume we have a patch embedding output (1, 256, 768); corresponding to (batch, num_patches, position_embedding). since we have 256 patches, then can't our network understand that each patch is in the position of its index value? Why do we have to explicitly define a position embedding for each patch?. Also, please kindly explain the implementation above I'm not sure I understand the mapping and why its initialised to zero</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-03T19:29:34.357Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-06T09:58:07.307Z
Comments.Text: the initialization is likely done somewhere else, the zeros in your linked code just reserve memory for the real parameters later.
Comments.UserDisplayName: N. Kiefer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-13T20:32:41.513Z
Comments.Text: Can you please put your **specific question** in the title?
Comments.UserDisplayName: nbro
Tags: <machine-learning><deep-learning><computer-vision><transformer><vision-transformer>
Title: How does Position embeddings work in Vision Transformer
UserDisplayName: a__ys
ThreadText: # How does Position embeddings work in Vision Transformer
  
  (Asked by: username_0 on 2023-03-30)
  
  I'm a bit confused how the position embedding in happened to each patch in the transformer. I thought Ideally we'd want each patch to have a value of (1, 2, 3, 4....) to describe the position of the patch in the image. but from the implementation here there do something like this:
   # positional embedding
              self.pos_embed = nn.Parameter(
                  torch.zeros(1, num_patches, embedding_dim)
              )
  
  Which is quite confusing because now we have some sort of mapping instead of just a value appended to each patch. Also, there is some sort of implicit position appended to the patch right? Assume we have a patch embedding output (1, 256, 768); corresponding to (batch, num_patches, position_embedding). since we have 256 patches, then can't our network understand that each patch is in the position of its index value? Why do we have to explicitly define a position embedding for each patch?. Also, please kindly explain the implementation above I'm not sure I understand the mapping and why its initialised to zero
  
  **Tags:** machine-learning, deep-learning, computer-vision, transformer, vision-transformer
  > **Comments:**
  > **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  > **username_2**: the initialization is likely done somewhere else, the zeros in your linked code just reserve memory for the real parameters later.  
  > **username_3**: Can you please put your **specific question** in the title?  
  
  ---
  
  ## 0 answers
  
  