Answers.Body: <p>It does not matter.</p>
  <p>Although, I can imagine a situation where it could matter a bit - when position embeddings are not learnt but calculated and fixed like in the original transformer (<em>Attention is all you need</em>). Then the attention mechanism may be slightly biased towards one or another position. But AFAIR in ViT the position embeddings are learnt. Not sure about other papers you mention.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-05T21:52:57.827Z
Answers.LastActivityDate: 2023-03-05T21:52:57.827Z
Answers.UserDisplayName: hans
Answers.Body: <p>It should not matter. To explain why, we need to understand how a transformer works.</p>
  <p>Transformers were originally designed for language models. They compute a self attention matrix, which is a fancy way of saying they input a sequence, and learn how every part of that sequence relates to every other part of the sequence.</p>
  <p>Since self attention is computed for every token with respect to every other token, transformers are completely agnostic to the order in which tokens are placed.</p>
  <p>For some applications this is a desirable property, but for others, we want to give the transformer some knowledge of the sequence order. This is done by adding positional embeddings. These are added to the sequence prior to being passed into the transformer, and they are used to encode sequence information.</p>
  <p>So to review:</p>
  <ol>
  <li>Vanilla transformers are totally agnostic to the sequence order.</li>
  <li>For applications where this is undesirable a positional encoding is added.</li>
  <li>The changing order that you observed should not matter.</li>
  </ol>
  <p><a href="https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/" rel="nofollow noreferrer">This</a> blog post may be helpful for further information.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-05T20:51:46.433Z
Answers.LastActivityDate: 2023-03-05T20:51:46.433Z
Answers.UserDisplayName: chessprogrammer
Body: <p>I am reading through the <a href="https://paperswithcode.com/method/vision-transformer" rel="nofollow noreferrer">Vision Transformer</a> paper and other related papers, such as <a href="https://arxiv.org/pdf/2012.12877.pdf" rel="nofollow noreferrer">DeiT</a> and <a href="https://arxiv.org/pdf/2203.12119.pdf" rel="nofollow noreferrer">Visual Prompt Tuning (VPT)</a>. I wonder if the position of the tokens that flow through the Transformer encode matters? Let's break it down, assuming we use ViT 16x16:</p>
  <ul>
  <li>The original Vision Transformer has 1 classifier <code>[CLS]</code> token, along with 16x16=196 image tokens. In general, we have 197 tokens overall, where the position is: <code>[[CLS], [Image Patches] x 196]</code></li>
  <li>DeiT further adds one Distillation token to the end of the sequence, so we have 198 overall: <code>[[CLS], [Image Patches] x 196, [Distill Token]]</code></li>
  <li>However, I notice a very strange thing with VPT that they add the Prompt token <strong>after the <code>[CLS]</code> but before the Image patches</strong>. So the position of the tokens in VPT is: <code>[[CLS], [Prompts] x N, [Image Patches] x 196]</code>.</li>
  </ul>
  <p>I wonder does such a thing matter? What if we change the position of these tokens, e.g., putting the Prompt tokens to the last?</p>
  
Tags: <computer-vision><transformer><vision-transformer>
Title: Does the position of the tokens in Vision Transformer matter?
UserDisplayName: Minh-Long Luu
ThreadText: # Does the position of the tokens in Vision Transformer matter?
  
  (Asked by: username_0 on 2023-01-08)
  
  I am reading through the [Vision Transformer](https://paperswithcode.com/method/vision-transformer) paper and other related papers, such as [DeiT](https://arxiv.org/pdf/2012.12877.pdf) and [Visual Prompt Tuning (VPT)](https://arxiv.org/pdf/2203.12119.pdf). I wonder if the position of the tokens that flow through the Transformer encode matters? Let's break it down, assuming we use ViT 16x16:
  
  * The original Vision Transformer has 1 classifier `[CLS]` token, along with 16x16=196 image tokens. In general, we have 197 tokens overall, where the position is: `[[CLS], [Image Patches] x 196]`
  * DeiT further adds one Distillation token to the end of the sequence, so we have 198 overall: `[[CLS], [Image Patches] x 196, [Distill Token]]`
  * However, I notice a very strange thing with VPT that they add the Prompt token **after the `[CLS]` but before the Image patches**. So the position of the tokens in VPT is: `[[CLS], [Prompts] x N, [Image Patches] x 196]`.
  
  I wonder does such a thing matter? What if we change the position of these tokens, e.g., putting the Prompt tokens to the last?
  
  **Tags:** computer-vision, transformer, vision-transformer
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-03-05. Score: 3 | Accepted answer
  
  It should not matter. To explain why, we need to understand how a transformer works.
  
  Transformers were originally designed for language models. They compute a self attention matrix, which is a fancy way of saying they input a sequence, and learn how every part of that sequence relates to every other part of the sequence.
  
  Since self attention is computed for every token with respect to every other token, transformers are completely agnostic to the order in which tokens are placed.
  
  For some applications this is a desirable property, but for others, we want to give the transformer some knowledge of the sequence order. This is done by adding positional embeddings. These are added to the sequence prior to being passed into the transformer, and they are used to encode sequence information.
  
  So to review:
  
  1. Vanilla transformers are totally agnostic to the sequence order.
  2. For applications where this is undesirable a positional encoding is added.
  3. The changing order that you observed should not matter.
  
  [This](https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/) blog post may be helpful for further information.
  
  
  ---
  
  ### Answer by username_1 on 2023-03-05. Score: 2
  
  It does not matter.
  
  Although, I can imagine a situation where it could matter a bit - when position embeddings are not learnt but calculated and fixed like in the original transformer (*Attention is all you need*). Then the attention mechanism may be slightly biased towards one or another position. But AFAIR in ViT the position embeddings are learnt. Not sure about other papers you mention.
  
  
  ---
  
  