<p>I'm a bit confused how the position embedding in happened to each patch in the transformer. I thought Ideally we'd want each patch to have a value of (1, 2, 3, 4....) to describe the position of the patch in the image. but from the implementation <a href="https://github.com/gupta-abhay/pytorch-vit/blob/main/vit/patch_embed.py" rel="nofollow noreferrer">here</a> there do something like this:</p>
<pre><code> # positional embedding
            self.pos_embed = nn.Parameter(
                torch.zeros(1, num_patches, embedding_dim)
            )
</code></pre>
<p>Which is quite confusing because now we have some sort of mapping instead of just a value appended to each patch. Also, there is some sort of implicit position appended to the patch right? Assume we have a patch embedding output (1, 256, 768); corresponding to (batch, num_patches, position_embedding). since we have 256 patches, then can't our network understand that each patch is in the position of its index value? Why do we have to explicitly define a position embedding for each patch?. Also, please kindly explain the implementation above I'm not sure I understand the mapping and why its initialised to zero</p>

CC BY-SA 4.0
2023-04-03T19:29:34.357Z
Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Community
CC BY-SA 4.0
2023-04-06T09:58:07.307Z
the initialization is likely done somewhere else, the zeros in your linked code just reserve memory for the real parameters later.
N. Kiefer
CC BY-SA 4.0
2023-04-13T20:32:41.513Z
Can you please put your **specific question** in the title?
nbro
<machine-learning><deep-learning><computer-vision><transformer><vision-transformer>
How does Position embeddings work in Vision Transformer
a__ys
# How does Position embeddings work in Vision Transformer

(Asked by: username_0 on 2023-03-30)

I'm a bit confused how the position embedding in happened to each patch in the transformer. I thought Ideally we'd want each patch to have a value of (1, 2, 3, 4....) to describe the position of the patch in the image. but from the implementation here there do something like this:
 # positional embedding
            self.pos_embed = nn.Parameter(
                torch.zeros(1, num_patches, embedding_dim)
            )

Which is quite confusing because now we have some sort of mapping instead of just a value appended to each patch. Also, there is some sort of implicit position appended to the patch right? Assume we have a patch embedding output (1, 256, 768); corresponding to (batch, num_patches, position_embedding). since we have 256 patches, then can't our network understand that each patch is in the position of its index value? Why do we have to explicitly define a position embedding for each patch?. Also, please kindly explain the implementation above I'm not sure I understand the mapping and why its initialised to zero

**Tags:** machine-learning, deep-learning, computer-vision, transformer, vision-transformer

> **Comments:**
> **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
> **username_2**: the initialization is likely done somewhere else, the zeros in your linked code just reserve memory for the real parameters later.  
> **username_3**: Can you please put your **specific question** in the title?  

---

## 0 answers

