Answers.Body: <h2>Introduction to Transformer Architectures</h2>
  <p>The original transformer architecture, as introduced by <a href="https://arxiv.org/pdf/1706.03762" rel="nofollow noreferrer">Vaswani et al.</a> in their paper &quot;Attention is all you need&quot;, encompasses both the encoder and decoder components as depicted in the original diagram. Here, the encoder processes input sequences to generate an encoded representation, while the decoder generates output sequences from this encoded representation. Although <a href="https://arxiv.org/pdf/1706.03762" rel="nofollow noreferrer">Vaswani et al.</a> referred to these components simply as &quot;Encoder&quot; and &quot;Decoder&quot; in the context of the sequence-to-sequence (seq2seq) task, subsequent terminology has evolved. In practice, the left block of the architecture is now commonly referred to as the <strong>Transformer Encoder</strong>, and the right block as the <strong>Transformer Decoder</strong>. This terminology has been adopted by various frameworks, including PyTorch, which provides specific classes such as <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html" rel="nofollow noreferrer"><code>TransformerEncoderLayer</code></a>, <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html" rel="nofollow noreferrer"><code>TransformerEncoder</code></a>, <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoderLayer.html" rel="nofollow noreferrer"><code>TransformerDecoderLayer</code></a>, and <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html" rel="nofollow noreferrer"><code>TransformerDecoder</code></a>. However, notice that this structural division is not unique to transformers but is a common design in various architectures like U-Net (<a href="https://link.springer.com/chapter/10.1007/978-3-319-24574-4_28" rel="nofollow noreferrer">Ronneberger et al., 2015</a>) and autoencoders (<a href="https://www.cs.cmu.edu/%7Ebhiksha/courses/deeplearning/Fall.2016/pdfs/Chap8_PDP86.pdf" rel="nofollow noreferrer">Rumelhard et al., 1986</a>), where the encoder captures input data hierarchies, and the decoder reconstructs outputs from these encoded states.</p>
  <p>The nomenclature around transformers has evolved, sometimes diverging from the initial definitions. For instance, subsequent works like those by <a href="https://arxiv.org/pdf/1801.10198" rel="nofollow noreferrer">Liu et al.</a> redefined a Transformer Decoder to only include a variant of the Multi-Head Attention mechanism, called the Memory-Compressed Attention layers, without the accompanying encoder and thus Cross Multi-Head Attention. This reinterpretation significantly shifts the architecture's function and nomenclature, influencing subsequent models like GPT (<a href="https://www.mikecaptain.com/resources/pdf/GPT-1.pdf" rel="nofollow noreferrer">Radford et al., 2018</a>), which adopt a decoder-only architecture under a new definition. Similarly, models such as BERT (Bidirectional Representations from Transformers, <a href="https://arxiv.org/pdf/1810.04805" rel="nofollow noreferrer">Devlin et. al, 2018</a>) and Vit (Vision Transformer, <a href="https://arxiv.org/pdf/2010.11929" rel="nofollow noreferrer">Dosovitskiy et al., 2020</a>) are described as employing only Transformer Encoders because they exclusively employs Multi-Head Self Attention mechanisms. These adaptations underscore a broader trend where &quot;transformer&quot; no longer strictly implies a seq2seq model, but rather any architecture leveraging repetitive attention mechanisms and feedforward networks, irrespective of the exact structure proposed by <a href="https://arxiv.org/pdf/1706.03762" rel="nofollow noreferrer">Vaswani et al</a>.</p>
  <h2>Is this whole thing a transformer?</h2>
  <p>Yes. However, to keep up with the times, I'd call it &quot;Transformer Encoder Decoder&quot;. The term &quot;Transformer&quot; now refers more broadly to any architecture that relies on the repetition of attention mechanisms (self, multi-head, masked, cross, etc.) and feedforward networks, meant to process input sequences of tokens. This definition encompasses models like BERT (<a href="https://arxiv.org/pdf/1810.04805" rel="nofollow noreferrer">Devlin et. al, 2018</a>), ViT (<a href="https://arxiv.org/pdf/2010.11929" rel="nofollow noreferrer">Dosovitskiy et al., 2020</a>), and GPT (<a href="https://www.mikecaptain.com/resources/pdf/GPT-1.pdf" rel="nofollow noreferrer">Radford et al., 2018</a>), which may not include both encoder and decoder components or perform seq2seq tasks.</p>
  <h2>What parts or what relationships between parts make it a transformer?</h2>
  <p>As I mentioned in the previous answer, originally a transformer was designed for seq2seq tasks using both encoder and decoder blocks centered around attention mechanisms. Modern interpretations, however, extend the definition to any model that incorporates these attention mechanisms within a repetitive architectural framework, even if the tasks and specific configurations vary. For instance, the Vit (<a href="https://arxiv.org/pdf/2010.11929" rel="nofollow noreferrer">Dosovitskiy et al., 2020</a>) has expanded the use of transformers to tasks like image classification, where the architecture processes a sequence of input tokens (image patches) but outputs a single label instead of a sequence. In this case, &quot;transformer&quot; still rely on their core mechanism of self-attention to integrate information across the input sequence, but the output is not necessarily a sequence. One could argue that the ViT still performs a seq2seq task, as it transforms the input sequence of tokens (image patches) into another sequence, which is then used to perform the final classification. However, this is a philosophical debate.</p>
  <h2>Equivalently, what aspects can I change before it becomes something else and not a transformer?</h2>
  <p>If you remove the attention layers, you would end up in a &quot;simple&quot; feedforward neural network (with some little variations, such as residual connections and layer normalizations).</p>
  <h2>If I only care about self-attention I suppose I don't need the right hand column. If I just keep the self-attention, is it still a transformer?</h2>
  <p>Yes, if you keep the self-attention mechanism but discard the right-hand column (i.e., the decoder component in the original transformer architecture), the resulting model can still be considered a transformer. This is exaclty what a Transformer Encoder is, such as BERT (<a href="https://arxiv.org/pdf/1810.04805" rel="nofollow noreferrer">Devlin et. al, 2018</a>) and ViT (<a href="https://arxiv.org/pdf/2010.11929" rel="nofollow noreferrer">Dosovitskiy et al., 2020</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-12T10:59:48.067Z
Answers.LastActivityDate: 2024-07-13T10:15:32.120Z
Answers.LastEditDate: 2024-07-13T10:15:32.120Z
Answers.UserDisplayName: frad
Body: <p>Which part of this is the transformer?</p>
  <p><a href="https://i.sstatic.net/qrbxN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qrbxN.png" alt="enter image description here" /></a></p>
  <p>Ok, the caption says the whole thing is the transformer, but that's back in 2017 when the paper was published. My question is about how the community uses the term &quot;transformer&quot; now.</p>
  <p>I'm not looking for an inline response to these questions. They are all a way of asking the same general thing.</p>
  <ul>
  <li>Is this whole thing a transformer?</li>
  <li>What parts or what relationships between parts make it a transformer?</li>
  <li>Equivalently, what aspects can I change before it becomes something else and not a transformer?</li>
  <li>If I only care about self-attention I suppose I don't need the right hand column. If I just keep the self-attention, is it still a transformer?</li>
  </ul>
  <p>Context about me is I've just become familiar with transformers and have not read much literature on them since this paper.</p>
  
Tags: <neural-networks><deep-learning><terminology><transformer>
Title: What part of the Vaswani et al. is the "transformer"?
UserDisplayName: Alexander Soare
ThreadText: # What part of the Vaswani et al. is the "transformer"?
  
  (Asked by: username_0 on 2021-04-14)
  
  Which part of this is the transformer?
  
  [![enter image description here](https://i.sstatic.net/qrbxN.png)](https://i.sstatic.net/qrbxN.png)
  
  Ok, the caption says the whole thing is the transformer, but that's back in 2017 when the paper was published. My question is about how the community uses the term "transformer" now.
  
  I'm not looking for an inline response to these questions. They are all a way of asking the same general thing.
  
  * Is this whole thing a transformer?
  * What parts or what relationships between parts make it a transformer?
  * Equivalently, what aspects can I change before it becomes something else and not a transformer?
  * If I only care about self-attention I suppose I don't need the right hand column. If I just keep the self-attention, is it still a transformer?
  
  Context about me is I've just become familiar with transformers and have not read much literature on them since this paper.
  
  **Tags:** neural-networks, deep-learning, terminology, transformer
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-12. Score: 0
  
  ## Introduction to Transformer Architectures
  
  The original transformer architecture, as introduced by [Vaswani et al.](https://arxiv.org/pdf/1706.03762) in their paper "Attention is all you need", encompasses both the encoder and decoder components as depicted in the original diagram. Here, the encoder processes input sequences to generate an encoded representation, while the decoder generates output sequences from this encoded representation. Although [Vaswani et al.](https://arxiv.org/pdf/1706.03762) referred to these components simply as "Encoder" and "Decoder" in the context of the sequence-to-sequence (seq2seq) task, subsequent terminology has evolved. In practice, the left block of the architecture is now commonly referred to as the **Transformer Encoder**, and the right block as the **Transformer Decoder**. This terminology has been adopted by various frameworks, including PyTorch, which provides specific classes such as [`TransformerEncoderLayer`](https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html), [`TransformerEncoder`](https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html), [`TransformerDecoderLayer`](https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoderLayer.html), and [`TransformerDecoder`](https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html). However, notice that this structural division is not unique to transformers but is a common design in various architectures like U-Net ([Ronneberger et al., 2015](https://link.springer.com/chapter/10.1007/978-3-319-24574-4_28)) and autoencoders ([Rumelhard et al., 1986](https://www.cs.cmu.edu/%7Ebhiksha/courses/deeplearning/Fall.2016/pdfs/Chap8_PDP86.pdf)), where the encoder captures input data hierarchies, and the decoder reconstructs outputs from these encoded states.
  
  The nomenclature around transformers has evolved, sometimes diverging from the initial definitions. For instance, subsequent works like those by [Liu et al.](https://arxiv.org/pdf/1801.10198) redefined a Transformer Decoder to only include a variant of the Multi-Head Attention mechanism, called the Memory-Compressed Attention layers, without the accompanying encoder and thus Cross Multi-Head Attention. This reinterpretation significantly shifts the architecture's function and nomenclature, influencing subsequent models like GPT ([Radford et al., 2018](https://www.mikecaptain.com/resources/pdf/GPT-1.pdf)), which adopt a decoder-only architecture under a new definition. Similarly, models such as BERT (Bidirectional Representations from Transformers, [Devlin et. al, 2018](https://arxiv.org/pdf/1810.04805)) and Vit (Vision Transformer, [Dosovitskiy et al., 2020](https://arxiv.org/pdf/2010.11929)) are described as employing only Transformer Encoders because they exclusively employs Multi-Head Self Attention mechanisms. These adaptations underscore a broader trend where "transformer" no longer strictly implies a seq2seq model, but rather any architecture leveraging repetitive attention mechanisms and feedforward networks, irrespective of the exact structure proposed by [Vaswani et al](https://arxiv.org/pdf/1706.03762).
  
  ## Is this whole thing a transformer?
  
  Yes. However, to keep up with the times, I'd call it "Transformer Encoder Decoder". The term "Transformer" now refers more broadly to any architecture that relies on the repetition of attention mechanisms (self, multi-head, masked, cross, etc.) and feedforward networks, meant to process input sequences of tokens. This definition encompasses models like BERT ([Devlin et. al, 2018](https://arxiv.org/pdf/1810.04805)), ViT ([Dosovitskiy et al., 2020](https://arxiv.org/pdf/2010.11929)), and GPT ([Radford et al., 2018](https://www.mikecaptain.com/resources/pdf/GPT-1.pdf)), which may not include both encoder and decoder components or perform seq2seq tasks.
  
  ## What parts or what relationships between parts make it a transformer?
  
  As I mentioned in the previous answer, originally a transformer was designed for seq2seq tasks using both encoder and decoder blocks centered around attention mechanisms. Modern interpretations, however, extend the definition to any model that incorporates these attention mechanisms within a repetitive architectural framework, even if the tasks and specific configurations vary. For instance, the Vit ([Dosovitskiy et al., 2020](https://arxiv.org/pdf/2010.11929)) has expanded the use of transformers to tasks like image classification, where the architecture processes a sequence of input tokens (image patches) but outputs a single label instead of a sequence. In this case, "transformer" still rely on their core mechanism of self-attention to integrate information across the input sequence, but the output is not necessarily a sequence. One could argue that the ViT still performs a seq2seq task, as it transforms the input sequence of tokens (image patches) into another sequence, which is then used to perform the final classification. However, this is a philosophical debate.
  
  ## Equivalently, what aspects can I change before it becomes something else and not a transformer?
  
  If you remove the attention layers, you would end up in a "simple" feedforward neural network (with some little variations, such as residual connections and layer normalizations).
  
  ## If I only care about self-attention I suppose I don't need the right hand column. If I just keep the self-attention, is it still a transformer?
  
  Yes, if you keep the self-attention mechanism but discard the right-hand column (i.e., the decoder component in the original transformer architecture), the resulting model can still be considered a transformer. This is exaclty what a Transformer Encoder is, such as BERT ([Devlin et. al, 2018](https://arxiv.org/pdf/1810.04805)) and ViT ([Dosovitskiy et al., 2020](https://arxiv.org/pdf/2010.11929)).
  
  
  ---
  
  