Answers.Body: <h1>Introduction</h1>
  <p>Large-language models (LLMs) have gained tons of popularity lately with the releases of ChatGPT, GPT-4, Bard, and more. All these LLMs are based on the transformer neural network architecture. The transformer architecture was first introduced in the paper &quot;<a href="https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html" rel="noreferrer">Attention is All You Need</a>&quot; by Google Brain in 2017. LLMs/GPT models use a variant of this architecture called de' decoder-only transformer'.</p>
  <p>The most popular variety of transformers are currently these GPT models. The only purpose of these models is to receive a prompt (an input) and predict the next token/word that comes after this input. Nothing more, nothing less.</p>
  <p><strong>Note:</strong> Not all large-language models use a transformer architecture. However, models such as GPT-3, ChatGPT, GPT-4 &amp; LaMDa use the (decoder-only) transformer architecture.</p>
  <h1>Overview of the (decoder-only) Transformer model</h1>
  <p>It is key first to understand the input and output of a transformer:</p>
  <ul>
  <li><strong>The input</strong> is a prompt (often referred to as context) fed into the transformer as a whole. There is no recurrence.</li>
  <li><strong>The output</strong> depends on the goal of the model. For GPT models, the output is a probability distribution of the next token/word that comes after the prompt. It outputs <strong>one</strong> prediction for the complete input.</li>
  </ul>
  <p>Next, it is essential to understand the key components that make up the decoder-only transformer architecture:</p>
  <ol>
  <li><strong>The embedding</strong>: the input of the transformer model is a prompt. This prompt needs to be embedded into something that the model can use.</li>
  <li><strong>The block(s)</strong>: This is the main source of complexity. Each block contains a masked multi-head attention submodule, a feedforward network, and several layer normalization operations. <em>Blocks are put in sequence</em> to make the model deeper.</li>
  <li><strong>The output</strong>: the output of the last block is fed through one more linear layer to obtain the final output of the model (a classification, a next word/token etc.)</li>
  </ol>
  <p>The following visualization gives an overview of the transformer architecture.</p>
  <p><a href="https://i.sstatic.net/bWnx0.png" rel="noreferrer"><img src="https://i.sstatic.net/bWnx0.png" alt="Overview of transformer architecture" /></a></p>
  <h1>Self-Attention Mechanism</h1>
  <p>Self-attention makes the transformer powerful. The intuition of self-attention is that the mechanism allows the model to focus on (attend to) the most relevant parts of the input. A single self-attention mechanism is called a <strong>head</strong>.</p>
  <p>The head works as follows. First, the input is fed into three separate linear layers. Two of those (the queries Q and the keys K) are multiplied, scaled, and turned into a probability distribution using a softmax activation function. Think of this probability distribution as describing which indices matter most for the output (i.e. which words in the prompt matter for the next word to be predicted). Finally, the output is multiplied with values V. This thus gives V * the importance of each of the tokens in V. A key observation is that the <em>learnable parameters</em> in the head are the three linear layers.</p>
  <p>The following figure gives an overview of the operations done in a head and an overview of how multi-head attention works.</p>
  <p><a href="https://i.sstatic.net/wH0ra.png" rel="noreferrer"><img src="https://i.sstatic.net/wH0ra.png" alt="Overview of attention mechanism" /></a></p>
  <h3>Multi-Head Attention</h3>
  <p>Multi-head attention is nothing more than several individual heads stacked on top of one another. The input to all heads is equivalent. However, each head has its own weights. After forwarding the input through all the heads, the output of the heads is concatenated and passed through a linear layer which brings the dimensionality back to the dimension of the initial input.</p>
  <h3>Masked Self-Attention</h3>
  <p>In the decoder-only transformer, masked self-attention is nothing more than sequence padding. The 'masking' term is a left-over of the original encoder-decoder transformer module in which the encoder could see the whole (original language) sentence, and the decoder could only see the first part of the sentence which was already translated. As such, they called it 'masking'.</p>
  <h1>Block</h1>
  <p>Each block contains a multi-head attention submodule, a feedforward network, 2 layer-normalization operations, and 2 skip connections.</p>
  <p><strong>The feedforward network</strong> is simply a multi-layer perceptron. In the original paper, the proposed feedforward module consisted of (1) a fully connected layer; (2) a ReLU activation; (3) another fully connected layer; and (4) a dropout layer.</p>
  <p><strong>The 'add &amp; norm' blocks</strong> get the output from the multi-head attention/feedforward submodule and add it to the input into those modules. After that, a layer normalization operation is performed. Adding the input and output of a submodule together is known as a <strong>skip-connection.</strong> As blocks can be put in sequence, the skip connections help tremendously reduce the problem of vanishing or exploding gradients. In other words, skip connections are necessary to ensure proper backpropagation of the gradients.</p>
  <h1>Positional Embedding</h1>
  <p>Transformers take in a complete prompt at once (in contrast to RNNs) and embed this as one big Tensor. As such, transformers do not know which word is at what position in the sentence. This is problematic as the following two sentences mean entirely different things, only dependent on the order of the words:</p>
  <blockquote>
  <p>The boy chased the bird with a butterfly net.</p>
  </blockquote>
  <blockquote>
  <p>The bird chased the boy with a butterfly net.</p>
  </blockquote>
  <p>To that end, a positional embedding is added to allow the model to deduce which word is where. Positional embeddings can be learned using any embedding layer from your favourite AI library. However, the original authors proposed a much more complicated method, which does not require learning any parameters. Please find an elaborate explanation (not mine) on the original positional embedding <a href="https://www.youtube.com/watch?v=dichIcUZfOw" rel="noreferrer">here</a>.</p>
  <h1>Output</h1>
  <p>After the prompt is forwarded through all the blocks sequentially, the output is forwarded through one final linear layer. This final linear layer maps the output of the model back to the size of the 'vocabulary'. I.e. if you want to predict the next letter in a message, it would map to 26 (letters) + additional stuff (such as .,-!? etc.).</p>
  <p>The output of the model is a probability distribution. For GPT models, the output is the probability of each token being the next token in the sequence.</p>
  <h1>Training</h1>
  <h3>Basic training</h3>
  <p>The basic training process consists of <a href="https://ai.stackexchange.com/questions/10623/what-is-self-supervised-learning-in-machine-learning">self-supervised learning</a>. Simply put, you gather lots of text, strip the last word from that text, feed it as input into the transformer, check if the prediction matches the word you cut off and backpropagate the error.</p>
  <p>Every text/sentence/book/webpage can be separated into several samples.</p>
  <pre><code>sample = [
      [&quot;This&quot;],
      [&quot;This&quot;, &quot;is&quot;],
      [&quot;This&quot;, &quot;is&quot;, &quot;a&quot;]] # padding is added until the max-sequence length is reached.
  targets = [&quot;is&quot;, &quot;a&quot;, &quot;sample&quot;]
  </code></pre>
  <h3>Fine-tuning / transfer-learning</h3>
  <p>After the first stage of training is completed, the model is now a large-language model. As in, it can predict the next word based on a context. However, through fine-tuning/transfer-learning the model can be adapted to better suit the needs of the final application.</p>
  <p>One of the key reasons why ChatGPT &amp; GPT-4 seem so ridiculously impressive is because of this second stage of training. In this stage, the following process is executed <em>many</em> times:</p>
  <ol>
  <li>The model is given a prompt and generates different answers</li>
  <li>The different answers are ranked by a human from best to worst.</li>
  <li>The scores of the different answers are backpropagated.</li>
  </ol>
  <p>However, transformer models can also be used for different tasks than language generation. They can, for example, be used for sentiment analysis. After doing the basic training, a transformer can be fine-tuned for sentiment analysis by removing the outgoing linear layer and replacing it with a different layer suitable for the task to be executed. Consequently, it can be trained in a supervised fashion on a custom (sentiment analysis) dataset.</p>
  <h1>Inference (answer generation)</h1>
  <p>Doing inference with a transformer is just like training. You insert a prompt and out comes the next word/classification/other.</p>
  <p>For GPT models, this means that the prompt is extended one word at a time. You insert the prompt, and out comes the first word of the answer. The first word of the answer is now added to the prompt, creating a new, slightly different prompt. This prompt is again forwarded through the model, giving the prediction of a new word.</p>
  <p>As the output is the probability for each token to be the next one, you can do several things during inference. For one, you can <strong>sample</strong> from the probability distribution. This induces some randomness into the algorithm. You can also take the token that has the highest probability; then, the model becomes deterministic.</p>
  <h1>How can a transformer do X?</h1>
  <p>The transformer can do X because it has seen enough examples of similar sentences in its training to give a satisfactory output to your prompt. However, for the question 'Why can a transformer <strong>not</strong> do X?' the answers are vastly more diverse.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-30T01:31:29.917Z
Answers.Text: @user253751 “finetuning” refers to any “late stage” training process. Typically this is done to do domain or style adaptation, like you suggest, but more recently the idea of finetuning on multitask instruction or human feedback data has taken off substantially (T0, flan-T5, GPT-3.5, ChatGPT).
Answers.UserDisplayName: Stella Biderman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-28T19:54:50.640Z
Answers.Text: I have not implemented the code for decoder only so I have 3questions: 
  
  1. so it doesnt use the triangular mask? I have heard from 2 sources which it does, but I dont get, as we only feed inputs and not the outputs(unlike original transformer),how triangular mask on input data makes sense?
  
  2. does why its called `decoder only`? the architecture seems much closer to encoder part of original transformer model, than its decoder part!! specially when the mask also not different than encoder of original.
  
  3. is it autoregressive or still can be autoencoder to output the outputs in one pass?
Answers.UserDisplayName: Farhang Amaji
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T13:02:09.113Z
Answers.Text: 1) could you specify what you mean with 'triangular mask'? 2. there are different names for it. It can be called an encoder-only transformer as well. I personally feel decoder only makes more sense, since there is little 'encoding' going on in most of its application fields. Different papers might refer to it as encoder-only or decoder-only transformer 3. Transformers are autoregressive. They predict/generate new data based on previous (generated) data.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-03T17:02:12.460Z
Answers.Text: In the example you provided with setting up the text for sequences, what role do the padding tokens and end of sequence tokens play?  If we're training on a novel, then would each sequence be a sentence or would ew train by an arbitrary context size (i.e. 50 tokens).  In the latter case, where would we add the EOS tokens?
Answers.UserDisplayName: matsuo_basho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-05T14:59:21.057Z
Answers.Text: Since the transformer is basically a feed-forward neural network, you need to add padding to having something to forward throughout the other remaining nodes for which you do not have input. If you have an input that is as long as the maximum content size, you do not have an EOS usually (as you are predicting the next word.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-11T16:18:09.187Z
Answers.Text: Thanks!  Related question - for training for text generation with a decoder model, are the inputs to the loss function the predicted sequence (length n) and for the target (ground truth) the word we're trying to predict?  Or is the target the whole sequence (length n) offset by one position, where the last element is the word we're trying to predict?
Answers.UserDisplayName: matsuo_basho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-23T19:28:30.460Z
Answers.LastActivityDate: 2023-04-29T07:00:28.607Z
Answers.LastEditDate: 2023-04-29T07:00:28.607Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>How does the (decoder-only) transformer architecture work which is used in impressive models such as GPT-4?</p>
  
Tags: <deep-learning><transformer><attention><gpt><large-language-models>
Title: How does the (decoder-only) transformer architecture work?
UserDisplayName: Robin van Hoorn
ThreadText: # How does the (decoder-only) transformer architecture work?
  
  (Asked by: username_0 on 2023-04-23)
  
  How does the (decoder-only) transformer architecture work which is used in impressive models such as GPT-4?
  
  **Tags:** deep-learning, transformer, attention, gpt, large-language-models
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-04-23. Score: 45 | Accepted answer
  
  # Introduction
  
  Large-language models (LLMs) have gained tons of popularity lately with the releases of ChatGPT, GPT-4, Bard, and more. All these LLMs are based on the transformer neural network architecture. The transformer architecture was first introduced in the paper "[Attention is All You Need](https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)" by Google Brain in 2017. LLMs/GPT models use a variant of this architecture called de' decoder-only transformer'.
  
  The most popular variety of transformers are currently these GPT models. The only purpose of these models is to receive a prompt (an input) and predict the next token/word that comes after this input. Nothing more, nothing less.
  
  **Note:** Not all large-language models use a transformer architecture. However, models such as GPT-3, ChatGPT, GPT-4 & LaMDa use the (decoder-only) transformer architecture.
  
  # Overview of the (decoder-only) Transformer model
  
  It is key first to understand the input and output of a transformer:
  
  * **The input** is a prompt (often referred to as context) fed into the transformer as a whole. There is no recurrence.
  * **The output** depends on the goal of the model. For GPT models, the output is a probability distribution of the next token/word that comes after the prompt. It outputs **one** prediction for the complete input.
  
  Next, it is essential to understand the key components that make up the decoder-only transformer architecture:
  
  1. **The embedding**: the input of the transformer model is a prompt. This prompt needs to be embedded into something that the model can use.
  2. **The block(s)**: This is the main source of complexity. Each block contains a masked multi-head attention submodule, a feedforward network, and several layer normalization operations. *Blocks are put in sequence* to make the model deeper.
  3. **The output**: the output of the last block is fed through one more linear layer to obtain the final output of the model (a classification, a next word/token etc.)
  
  The following visualization gives an overview of the transformer architecture.
  
  [![Overview of transformer architecture](https://i.sstatic.net/bWnx0.png)](https://i.sstatic.net/bWnx0.png)
  
  # Self-Attention Mechanism
  
  Self-attention makes the transformer powerful. The intuition of self-attention is that the mechanism allows the model to focus on (attend to) the most relevant parts of the input. A single self-attention mechanism is called a **head**.
  
  The head works as follows. First, the input is fed into three separate linear layers. Two of those (the queries Q and the keys K) are multiplied, scaled, and turned into a probability distribution using a softmax activation function. Think of this probability distribution as describing which indices matter most for the output (i.e. which words in the prompt matter for the next word to be predicted). Finally, the output is multiplied with values V. This thus gives V \* the importance of each of the tokens in V. A key observation is that the *learnable parameters* in the head are the three linear layers.
  
  The following figure gives an overview of the operations done in a head and an overview of how multi-head attention works.
  
  [![Overview of attention mechanism](https://i.sstatic.net/wH0ra.png)](https://i.sstatic.net/wH0ra.png)
  
  ### Multi-Head Attention
  
  Multi-head attention is nothing more than several individual heads stacked on top of one another. The input to all heads is equivalent. However, each head has its own weights. After forwarding the input through all the heads, the output of the heads is concatenated and passed through a linear layer which brings the dimensionality back to the dimension of the initial input.
  
  ### Masked Self-Attention
  
  In the decoder-only transformer, masked self-attention is nothing more than sequence padding. The 'masking' term is a left-over of the original encoder-decoder transformer module in which the encoder could see the whole (original language) sentence, and the decoder could only see the first part of the sentence which was already translated. As such, they called it 'masking'.
  
  # Block
  
  Each block contains a multi-head attention submodule, a feedforward network, 2 layer-normalization operations, and 2 skip connections.
  
  **The feedforward network** is simply a multi-layer perceptron. In the original paper, the proposed feedforward module consisted of (1) a fully connected layer; (2) a ReLU activation; (3) another fully connected layer; and (4) a dropout layer.
  
  **The 'add & norm' blocks** get the output from the multi-head attention/feedforward submodule and add it to the input into those modules. After that, a layer normalization operation is performed. Adding the input and output of a submodule together is known as a **skip-connection.** As blocks can be put in sequence, the skip connections help tremendously reduce the problem of vanishing or exploding gradients. In other words, skip connections are necessary to ensure proper backpropagation of the gradients.
  
  # Positional Embedding
  
  Transformers take in a complete prompt at once (in contrast to RNNs) and embed this as one big Tensor. As such, transformers do not know which word is at what position in the sentence. This is problematic as the following two sentences mean entirely different things, only dependent on the order of the words:
  
  > The boy chased the bird with a butterfly net.
  
  > The bird chased the boy with a butterfly net.
  
  To that end, a positional embedding is added to allow the model to deduce which word is where. Positional embeddings can be learned using any embedding layer from your favourite AI library. However, the original authors proposed a much more complicated method, which does not require learning any parameters. Please find an elaborate explanation (not mine) on the original positional embedding [here](https://www.youtube.com/watch?v=dichIcUZfOw).
  
  # Output
  
  After the prompt is forwarded through all the blocks sequentially, the output is forwarded through one final linear layer. This final linear layer maps the output of the model back to the size of the 'vocabulary'. I.e. if you want to predict the next letter in a message, it would map to 26 (letters) + additional stuff (such as .,-!? etc.).
  
  The output of the model is a probability distribution. For GPT models, the output is the probability of each token being the next token in the sequence.
  
  # Training
  
  ### Basic training
  
  The basic training process consists of [self-supervised learning](https://ai.stackexchange.com/questions/10623/what-is-self-supervised-learning-in-machine-learning). Simply put, you gather lots of text, strip the last word from that text, feed it as input into the transformer, check if the prediction matches the word you cut off and backpropagate the error.
  
  Every text/sentence/book/webpage can be separated into several samples.
  
  ```
  sample = [
      ["This"],
      ["This", "is"],
      ["This", "is", "a"]] # padding is added until the max-sequence length is reached.
  targets = ["is", "a", "sample"]
  
  ```
  
  ### Fine-tuning / transfer-learning
  
  After the first stage of training is completed, the model is now a large-language model. As in, it can predict the next word based on a context. However, through fine-tuning/transfer-learning the model can be adapted to better suit the needs of the final application.
  
  One of the key reasons why ChatGPT & GPT-4 seem so ridiculously impressive is because of this second stage of training. In this stage, the following process is executed *many* times:
  
  1. The model is given a prompt and generates different answers
  2. The different answers are ranked by a human from best to worst.
  3. The scores of the different answers are backpropagated.
  
  However, transformer models can also be used for different tasks than language generation. They can, for example, be used for sentiment analysis. After doing the basic training, a transformer can be fine-tuned for sentiment analysis by removing the outgoing linear layer and replacing it with a different layer suitable for the task to be executed. Consequently, it can be trained in a supervised fashion on a custom (sentiment analysis) dataset.
  
  # Inference (answer generation)
  
  Doing inference with a transformer is just like training. You insert a prompt and out comes the next word/classification/other.
  
  For GPT models, this means that the prompt is extended one word at a time. You insert the prompt, and out comes the first word of the answer. The first word of the answer is now added to the prompt, creating a new, slightly different prompt. This prompt is again forwarded through the model, giving the prediction of a new word.
  
  As the output is the probability for each token to be the next one, you can do several things during inference. For one, you can **sample** from the probability distribution. This induces some randomness into the algorithm. You can also take the token that has the highest probability; then, the model becomes deterministic.
  
  # How can a transformer do X?
  
  The transformer can do X because it has seen enough examples of similar sentences in its training to give a satisfactory output to your prompt. However, for the question 'Why can a transformer **not** do X?' the answers are vastly more diverse.
  
  > **Comments:**
  > **username_1**: @user253751 “finetuning” refers to any “late stage” training process. Typically this is done to do domain or style adaptation, like you suggest, but more recently the idea of finetuning on multitask instruction or human feedback data has taken off substantially (T0, flan-T5, GPT-3.5, ChatGPT).  
  > **username_2**: I have not implemented the code for decoder only so I have 3questions:   1. so it doesnt use the triangular mask? I have heard from 2 sources which it does, but I dont get, as we only feed inputs and not the outputs(unlike original transformer),how triangular mask on input data makes sense?  2. does why its called `decoder only`? the architecture seems much closer to encoder part of original transformer model, than its decoder part!! specially when the mask also not different than encoder of original.  3. is it autoregressive or still can be autoencoder to output the outputs in one pass?  
  > **username_0**: 1) could you specify what you mean with 'triangular mask'? 2. there are different names for it. It can be called an encoder-only transformer as well. I personally feel decoder only makes more sense, since there is little 'encoding' going on in most of its application fields. Different papers might refer to it as encoder-only or decoder-only transformer 3. Transformers are autoregressive. They predict/generate new data based on previous (generated) data.  
  > **username_3**: In the example you provided with setting up the text for sequences, what role do the padding tokens and end of sequence tokens play?  If we're training on a novel, then would each sequence be a sentence or would ew train by an arbitrary context size (i.e. 50 tokens).  In the latter case, where would we add the EOS tokens?  
  > **username_0**: Since the transformer is basically a feed-forward neural network, you need to add padding to having something to forward throughout the other remaining nodes for which you do not have input. If you have an input that is as long as the maximum content size, you do not have an EOS usually (as you are predicting the next word.  
  > **username_3**: Thanks!  Related question - for training for text generation with a decoder model, are the inputs to the loss function the predicted sequence (length n) and for the target (ground truth) the word we're trying to predict?  Or is the target the whole sequence (length n) offset by one position, where the last element is the word we're trying to predict?  
  
  ---
  
  