<p>The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs.</p>
<p>The main part of the <strong>Vision Transformer</strong> are the self-attention layers.</p>
<p>The architecture proposed in the paper <a href="https://arxiv.org/abs/2010.11929" rel="nofollow noreferrer">An Image is Worth 16x16 Words</a> treats each 16x16 as a word in the sentence. There is a convolutional layer (with <code>kernel_size</code>=16 and  <code>stride</code> 16) that transforms the input patches into tokens as in NLP problem, and then these tokens are propagated through multiple layers.</p>
<p><a href="https://i.sstatic.net/xhb3S.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xhb3S.png" alt="enter image description here" /></a></p>
<p>Each Transformer encoder is a standard Transformer block, consisting of the:</p>
<ul>
<li>Multihead self-attention layer that transforms tokens into keys, queries and values</li>
<li><code>Feedforward layer</code> acting on each token independently (pointwise nonlinearity)</li>
<li><code>LayerNormalization</code> modules between them.</li>
</ul>
<p><a href="https://i.sstatic.net/uxwr4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uxwr4.png" alt="enter image description here" /></a></p>
<p>Each image is treated as a sentence or chunk of text.
The main idea and advantage of self-attention layers is the ability to collect the global context of the given data sample, whereas CNNs are restricted to a neighboorhood of the given pixel (and can have a global understanding of the data after a sufficient number of convolutional layers).</p>
<p>If you are inexperienced with transformers, I recommend reading <a href="https://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">this blog</a> as an easily accessible and comprehensive introduction to Transformers.</p>

CC BY-SA 4.0
2021-10-22T19:48:02.147Z
2021-10-22T19:48:02.147Z
spiridon_the_sun_rotator
<p>Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous deep learning-based methods that are used in deepfake detection, as I found in my study.</p>
<p>I was able to find that CNNs, RNNs and LSTMs are multilayered neural networks, but I found very little about the neural network layers in a Vision Transformer. (Like a typical CNN has an input layer, pooling layer, and a fully connected layer, and finally an output layer. RNN has an input layer, multiple hidden layers and an output layer.)</p>
<p>So, what are the <strong>main neural network layers</strong> in a <strong>Vision Transformer</strong>?</p>

<neural-networks><deep-learning><transformer><layers><vision-transformer>
What are the major layers in a Vision Transformer?
Pawara Siriwardhane
# What are the major layers in a Vision Transformer?

(Asked by: username_0 on 2021-10-22)

Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous deep learning-based methods that are used in deepfake detection, as I found in my study.

I was able to find that CNNs, RNNs and LSTMs are multilayered neural networks, but I found very little about the neural network layers in a Vision Transformer. (Like a typical CNN has an input layer, pooling layer, and a fully connected layer, and finally an output layer. RNN has an input layer, multiple hidden layers and an output layer.)

So, what are the **main neural network layers** in a **Vision Transformer**?

**Tags:** neural-networks, deep-learning, transformer, layers, vision-transformer


---

## 1 answers

### Answer by username_1 on 2021-10-22. Score: 4 | Accepted answer

The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs.

The main part of the **Vision Transformer** are the self-attention layers.

The architecture proposed in the paper [An Image is Worth 16x16 Words](https://arxiv.org/abs/2010.11929) treats each 16x16 as a word in the sentence. There is a convolutional layer (with `kernel_size`=16 and `stride` 16) that transforms the input patches into tokens as in NLP problem, and then these tokens are propagated through multiple layers.

[![enter image description here](https://i.sstatic.net/xhb3S.png)](https://i.sstatic.net/xhb3S.png)

Each Transformer encoder is a standard Transformer block, consisting of the:

* Multihead self-attention layer that transforms tokens into keys, queries and values
* `Feedforward layer` acting on each token independently (pointwise nonlinearity)
* `LayerNormalization` modules between them.

[![enter image description here](https://i.sstatic.net/uxwr4.png)](https://i.sstatic.net/uxwr4.png)

Each image is treated as a sentence or chunk of text.
The main idea and advantage of self-attention layers is the ability to collect the global context of the given data sample, whereas CNNs are restricted to a neighboorhood of the given pixel (and can have a global understanding of the data after a sufficient number of convolutional layers).

If you are inexperienced with transformers, I recommend reading [this blog](https://jalammar.github.io/illustrated-transformer/) as an easily accessible and comprehensive introduction to Transformers.


---

