Answers.Body: <p>The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs.</p>
  <p>The main part of the <strong>Vision Transformer</strong> are the self-attention layers.</p>
  <p>The architecture proposed in the paper <a href="https://arxiv.org/abs/2010.11929" rel="nofollow noreferrer">An Image is Worth 16x16 Words</a> treats each 16x16 as a word in the sentence. There is a convolutional layer (with <code>kernel_size</code>=16 and  <code>stride</code> 16) that transforms the input patches into tokens as in NLP problem, and then these tokens are propagated through multiple layers.</p>
  <p><a href="https://i.sstatic.net/xhb3S.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xhb3S.png" alt="enter image description here" /></a></p>
  <p>Each Transformer encoder is a standard Transformer block, consisting of the:</p>
  <ul>
  <li>Multihead self-attention layer that transforms tokens into keys, queries and values</li>
  <li><code>Feedforward layer</code> acting on each token independently (pointwise nonlinearity)</li>
  <li><code>LayerNormalization</code> modules between them.</li>
  </ul>
  <p><a href="https://i.sstatic.net/uxwr4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uxwr4.png" alt="enter image description here" /></a></p>
  <p>Each image is treated as a sentence or chunk of text.
  The main idea and advantage of self-attention layers is the ability to collect the global context of the given data sample, whereas CNNs are restricted to a neighboorhood of the given pixel (and can have a global understanding of the data after a sufficient number of convolutional layers).</p>
  <p>If you are inexperienced with transformers, I recommend reading <a href="https://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">this blog</a> as an easily accessible and comprehensive introduction to Transformers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-22T19:48:02.147Z
Answers.LastActivityDate: 2021-10-22T19:48:02.147Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous deep learning-based methods that are used in deepfake detection, as I found in my study.</p>
  <p>I was able to find that CNNs, RNNs and LSTMs are multilayered neural networks, but I found very little about the neural network layers in a Vision Transformer. (Like a typical CNN has an input layer, pooling layer, and a fully connected layer, and finally an output layer. RNN has an input layer, multiple hidden layers and an output layer.)</p>
  <p>So, what are the <strong>main neural network layers</strong> in a <strong>Vision Transformer</strong>?</p>
  
Tags: <neural-networks><deep-learning><transformer><layers><vision-transformer>
Title: What are the major layers in a Vision Transformer?
UserDisplayName: Pawara Siriwardhane
ThreadText: # What are the major layers in a Vision Transformer?
  
  (Asked by: username_0 on 2021-10-22)
  
  Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous deep learning-based methods that are used in deepfake detection, as I found in my study.
  
  I was able to find that CNNs, RNNs and LSTMs are multilayered neural networks, but I found very little about the neural network layers in a Vision Transformer. (Like a typical CNN has an input layer, pooling layer, and a fully connected layer, and finally an output layer. RNN has an input layer, multiple hidden layers and an output layer.)
  
  So, what are the **main neural network layers** in a **Vision Transformer**?
  
  **Tags:** neural-networks, deep-learning, transformer, layers, vision-transformer
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-10-22. Score: 4 | Accepted answer
  
  The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs.
  
  The main part of the **Vision Transformer** are the self-attention layers.
  
  The architecture proposed in the paper [An Image is Worth 16x16 Words](https://arxiv.org/abs/2010.11929) treats each 16x16 as a word in the sentence. There is a convolutional layer (with `kernel_size`=16 and `stride` 16) that transforms the input patches into tokens as in NLP problem, and then these tokens are propagated through multiple layers.
  
  [![enter image description here](https://i.sstatic.net/xhb3S.png)](https://i.sstatic.net/xhb3S.png)
  
  Each Transformer encoder is a standard Transformer block, consisting of the:
  
  * Multihead self-attention layer that transforms tokens into keys, queries and values
  * `Feedforward layer` acting on each token independently (pointwise nonlinearity)
  * `LayerNormalization` modules between them.
  
  [![enter image description here](https://i.sstatic.net/uxwr4.png)](https://i.sstatic.net/uxwr4.png)
  
  Each image is treated as a sentence or chunk of text.
  The main idea and advantage of self-attention layers is the ability to collect the global context of the given data sample, whereas CNNs are restricted to a neighboorhood of the given pixel (and can have a global understanding of the data after a sufficient number of convolutional layers).
  
  If you are inexperienced with transformers, I recommend reading [this blog](https://jalammar.github.io/illustrated-transformer/) as an easily accessible and comprehensive introduction to Transformers.
  
  
  ---
  
  