Answers.Body: <blockquote>
  <p>However, for me, it would be more natural not to create a special token, but perform 1d Global Pooling in the end, and attach an nn.Linear(embedding_dim, num_classes) as more conventional CV approach.</p>
  </blockquote>
  <p>Surprisingly, this approach was experimented in this new paper <code>Better plain ViT baselines for ImageNet-1k</code> and implemented as default in the <code>torch-vit</code> repository.</p>
  <p>Along with other trivial improvements, the authors actually outperform other methods such as DieT or ViT with strong data augmentation.  The modification you described provides 1.8% accuracy gain in the ablation study.</p>
  <p>The paper is essentially 1.5 pages so I definitely recommend having a look!</p>
  <p>link: <a href="https://arxiv.org/abs/2205.01580" rel="nofollow noreferrer">https://arxiv.org/abs/2205.01580</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-18T01:31:26.587Z
Answers.LastActivityDate: 2022-11-18T01:31:26.587Z
Answers.UserDisplayName: krenerd
Answers.Body: <p>In their <a href="https://github.com/google-research/vision_transformer" rel="nofollow noreferrer">official repository</a>, the author confirmed in the issue that the cls is not really important for the ViT, but they wanted to keep the ViT to be exactly the same with the NLP-Transformer, so they added it.</p>
  <p>Issue link: <a href="https://github.com/google-research/vision_transformer/issues/61#issuecomment-802233921" rel="nofollow noreferrer">https://github.com/google-research/vision_transformer/issues/61#issuecomment-802233921</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-10T15:12:55.997Z
Answers.LastActivityDate: 2022-03-10T15:12:55.997Z
Answers.UserDisplayName: CuCaRot
Body: <p>In the <a href="https://arxiv.org/abs/2010.11929" rel="noreferrer">famous work on the Visual Transformers</a>, the image is split into patches of a certain size (say 16x16), and these patches are treated as tokens in the NLP tasks. In order to perform classification, a <strong>CLS</strong> token is added at the beginning of the resulting sequence:
  <span class="math-container">$$
  [\textbf{x}_{class}, \textbf{x}_{p}^{1}, \ldots, \textbf{x}_{p}^{N}]
  ,$$</span>
  where <span class="math-container">$ \textbf{x}_{p}^{i}$</span> are image patches. There multiple layers in the architecture and the state of the <strong>CLS</strong> token on the output layer is used for classification.</p>
  <p>I think this architectural solution is done in the spirit of NLP problems (BERT in particular). However, for me, it would be more natural not to create a special token, but perform <em>1d Global Pooling</em> in the end, and attach an <code>nn.Linear(embedding_dim, num_classes)</code> as more conventional CV approach.</p>
  <p>Why it is not done in this way? Or is there some intuition or evidence that this would perform worse than the approach used in the paper?</p>
  
Tags: <computer-vision><classification><papers><transformer><vision-transformer>
Title: Why class embedding token is added to the Visual Transformer?
UserDisplayName: spiridon_the_sun_rotator
ThreadText: # Why class embedding token is added to the Visual Transformer?
  
  (Asked by: username_0 on 2021-06-19)
  
  In the [famous work on the Visual Transformers](https://arxiv.org/abs/2010.11929), the image is split into patches of a certain size (say 16x16), and these patches are treated as tokens in the NLP tasks. In order to perform classification, a **CLS** token is added at the beginning of the resulting sequence:
  $$
  [\textbf{x}\_{class}, \textbf{x}\_{p}^{1}, \ldots, \textbf{x}\_{p}^{N}]
  ,$$
  where $ \textbf{x}\_{p}^{i}$ are image patches. There multiple layers in the architecture and the state of the **CLS** token on the output layer is used for classification.
  
  I think this architectural solution is done in the spirit of NLP problems (BERT in particular). However, for me, it would be more natural not to create a special token, but perform *1d Global Pooling* in the end, and attach an `nn.Linear(embedding_dim, num_classes)` as more conventional CV approach.
  
  Why it is not done in this way? Or is there some intuition or evidence that this would perform worse than the approach used in the paper?
  
  **Tags:** computer-vision, classification, papers, transformer, vision-transformer
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-03-10. Score: 3 | Accepted answer
  
  In their [official repository](https://github.com/google-research/vision_transformer), the author confirmed in the issue that the cls is not really important for the ViT, but they wanted to keep the ViT to be exactly the same with the NLP-Transformer, so they added it.
  
  Issue link: <https://github.com/google-research/vision_transformer/issues/61#issuecomment-802233921>
  
  
  ---
  
  ### Answer by username_1 on 2022-11-18. Score: 0
  
  > However, for me, it would be more natural not to create a special token, but perform 1d Global Pooling in the end, and attach an nn.Linear(embedding\_dim, num\_classes) as more conventional CV approach.
  
  Surprisingly, this approach was experimented in this new paper `Better plain ViT baselines for ImageNet-1k` and implemented as default in the `torch-vit` repository.
  
  Along with other trivial improvements, the authors actually outperform other methods such as DieT or ViT with strong data augmentation. The modification you described provides 1.8% accuracy gain in the ablation study.
  
  The paper is essentially 1.5 pages so I definitely recommend having a look!
  
  link: <https://arxiv.org/abs/2205.01580>
  
  
  ---
  
  