Answers.Body: <p>In Machine Learning &quot;embedding&quot; means taking some set of raw inputs (like natural language tokens in NLP or image patches in your example) and converting them to vectors somehow. The embeddings usually have some interesting dot-product structure between vectors (like in <a href="https://en.wikipedia.org/wiki/Word2vec" rel="nofollow noreferrer">word2vec</a> for example). The Transformer machinery then uses this embedding in the dot-product attention pipeline. The dimension of the embedding <span class="math-container">$D$</span> should stay constant throughout the transformer blocks due to ResNet skip connections.</p>
  <p>The simplest idea in the case of the image patches would be to just take all the channels of all the pixels and treat them as a single vector. For example, if you've got (16, 16, 3) patches then you'll have 768-dimensional &quot;embeddings&quot;. The problem with such a naive &quot;embedding&quot; is that dot-products between them won't make much sense. So we also multiply these vectors by a trainable matrix <span class="math-container">$W$</span> and add a bias vector.</p>
  <p>For example, if you've got (16, 16, 3) patches and the transformer downstream uses <span class="math-container">$D=128$</span> dimensional embeddings, then you first flatten the patches into 16 * 16 * 3 = 768-dimensional vectors and then multiply by a 768<span class="math-container">$\times$</span>128  matrix and add a 128-dimensional bias vector.</p>
  <p>Looking at the code, it seems that the authors kept improving on that idea by adding one or several early convolutions with nonlinearities (the <code>conv_stem</code> branches in the code). The simplest execution branch before these embellishments seems to be <a href="https://github.com/gupta-abhay/pytorch-vit/blob/3fb85bf81a8a14cf4652f507e0156e0c95195b12/vit/patch_embed.py#L176" rel="nofollow noreferrer">here</a>. And the matrix multiplication I've been talking about <a href="https://github.com/gupta-abhay/pytorch-vit/blob/3fb85bf81a8a14cf4652f507e0156e0c95195b12/vit/patch_embed.py#L78" rel="nofollow noreferrer">is here</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-24T13:48:23.613Z
Answers.Text: Thank you for the nice information and for even referring to code implementation!
Answers.UserDisplayName: Hesham Eraqi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-25T23:13:42.950Z
Answers.Text: Isn't this embedding layer redundant since we also use projection matrices (linear)? I mean embedding + project is just a composition of two linear layers.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-08T08:31:23.447Z
Answers.Text: @adosar it would be redundant if we didn't have the nonlinear transformations in between embedding and projection.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-16T15:52:15.033Z
Answers.Text: @Kostya There is no non-linearity between the embedding and the $Q, K, V$ matrices.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-01T16:00:13.697Z
Answers.LastActivityDate: 2021-12-01T16:00:13.697Z
Answers.UserDisplayName: Kostya
Body: <p>I get the part from <a href="https://arxiv.org/abs/2010.11929" rel="nofollow noreferrer">the paper</a> where the image is split into <code>P</code> say <code>16x16</code> (smaller images) patches and then you have to <code>Flatten</code> the 3-D (16,16,3) patch to pass it into a <code>Linear</code> layer to get what they call &quot;Liner Projection&quot;. After passing from the Linear layer, the patches will be vectors but with some &quot;meaning&quot; to them.</p>
  <p>Can someone please explain how the two types of embeddings are working?</p>
  <p>I visited <a href="https://github.com/gupta-abhay/pytorch-vit/" rel="nofollow noreferrer">this implementation on github</a>, looked at the code too and looked like a maze to me.</p>
  <p>If someone could just explain how these embeddings are working in laymen's terms, I'll look at the code again and understand.</p>
  
Tags: <neural-networks><computer-vision><transformer><embeddings><vision-transformer>
Title: How does the embeddings work in vision transformer from paper?
UserDisplayName: Deshwal
ThreadText: # How does the embeddings work in vision transformer from paper?
  
  (Asked by: username_0 on 2021-01-26)
  
  I get the part from [the paper](https://arxiv.org/abs/2010.11929) where the image is split into `P` say `16x16` (smaller images) patches and then you have to `Flatten` the 3-D (16,16,3) patch to pass it into a `Linear` layer to get what they call "Liner Projection". After passing from the Linear layer, the patches will be vectors but with some "meaning" to them.
  
  Can someone please explain how the two types of embeddings are working?
  
  I visited [this implementation on github](https://github.com/gupta-abhay/pytorch-vit/), looked at the code too and looked like a maze to me.
  
  If someone could just explain how these embeddings are working in laymen's terms, I'll look at the code again and understand.
  
  **Tags:** neural-networks, computer-vision, transformer, embeddings, vision-transformer
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-01. Score: 2
  
  In Machine Learning "embedding" means taking some set of raw inputs (like natural language tokens in NLP or image patches in your example) and converting them to vectors somehow. The embeddings usually have some interesting dot-product structure between vectors (like in [word2vec](https://en.wikipedia.org/wiki/Word2vec) for example). The Transformer machinery then uses this embedding in the dot-product attention pipeline. The dimension of the embedding $D$ should stay constant throughout the transformer blocks due to ResNet skip connections.
  
  The simplest idea in the case of the image patches would be to just take all the channels of all the pixels and treat them as a single vector. For example, if you've got (16, 16, 3) patches then you'll have 768-dimensional "embeddings". The problem with such a naive "embedding" is that dot-products between them won't make much sense. So we also multiply these vectors by a trainable matrix $W$ and add a bias vector.
  
  For example, if you've got (16, 16, 3) patches and the transformer downstream uses $D=128$ dimensional embeddings, then you first flatten the patches into 16 \* 16 \* 3 = 768-dimensional vectors and then multiply by a 768$\times$128 matrix and add a 128-dimensional bias vector.
  
  Looking at the code, it seems that the authors kept improving on that idea by adding one or several early convolutions with nonlinearities (the `conv_stem` branches in the code). The simplest execution branch before these embellishments seems to be [here](https://github.com/gupta-abhay/pytorch-vit/blob/3fb85bf81a8a14cf4652f507e0156e0c95195b12/vit/patch_embed.py#L176). And the matrix multiplication I've been talking about [is here](https://github.com/gupta-abhay/pytorch-vit/blob/3fb85bf81a8a14cf4652f507e0156e0c95195b12/vit/patch_embed.py#L78)
  
  > **Comments:**
  > **username_2**: Thank you for the nice information and for even referring to code implementation!  
  > **username_3**: Isn't this embedding layer redundant since we also use projection matrices (linear)? I mean embedding + project is just a composition of two linear layers.  
  > **username_1**: @adosar it would be redundant if we didn't have the nonlinear transformations in between embedding and projection.  
  > **username_3**: @Kostya There is no non-linearity between the embedding and the $Q, K, V$ matrices.  
  
  ---
  
  