In this video, we'll study the decoder architecture. An example of a popular decoder-only architecture is GPT-2. In order to understand how decoders work, we recommend taking a look at the video regarding encoders: they're extremely similar to decoders. One can use a decoder for most of the same tasks as an encoder, albeit with, generally, a little loss of performance. Let's take the same approach we have taken with the encoder to try and understand the architectural differences between an encoder and a decoder. We'll use a small example, using three words. We pass them through the decoder. We retrieve a numerical representation of each word. Here, for example, the decoder converts the three words “Welcome to NYC” in these three sequences of numbers. The decoder outputs exactly one sequence of numbers per input word. This numerical representation can also be called a "Feature vector", or "Feature tensor". Let's dive in this representation. It contains one vector per word that was passed through the decoder. Each of these vector is a numerical representation of the word in question. The dimension of that vector is defined by the architecture of the model. Where the decoder differs from the encoder is principally with its self-attention mechanism. It's using what is called "masked self-attention". Here for example, if we focus on the word "to", we'll see that its vector is absolutely unmodified by the "NYC" word. That's because all the words on the right (also known as the right context) of the word is masked. Rather than benefitting from all the words on the left and right, I.e., the bidirectional context, decoders only have access to the words on their left. The masked self-attention mechanism differs from the self-attention mechanism by using an additional mask to hide the context on either side of the word: the word's numerical representation will not be affected by the words in the hidden context. So when should one use a decoder? Decoders, like encoders, can be used as standalone models. As they generate a numerical representation, they can also be used in a wide variety of tasks. However, the strength of a decoder lies in the way a word has access to its left context. The decoders, having only access to their left context, are inherently good at text generation: the ability to generate a word, or a sequence of words, given a known sequence of words. In NLP, this is known as Causal Language Modeling. Let's look at an example. Here's an example of how causal language modeling works: we start with an initial word, which is "My". We use this as input for the decoder. The model outputs a vectors of dimension 768. This vector contains information about the sequence, which is here a single word, or word. We apply a small transformation to that vector so that it maps to all the words known by the model (mapping which we'll see later, called a language modeling head). We identify that the model believes the most probable following word is "name". We then take that new word, and add it to the initial sequence. From "My", we are now at "My name". This is where the "autoregressive" aspect comes in. Auto-regressive models re-use their past outputs as inputs in the following steps. Once again, we do that the exact same operation: we cast that sequence through the decoder, and retrieve the most probable following word. In this case, it is the word "is". We repeat the operation until we're satisfied. Starting from a single word, we've now generated a full sentence. We decide to stop there, but we could continue for a while; GPT-2, for example, has a maximum context size of 1024. We could eventually generate up to 1024 words, and the decoder would still have some memory of the first words of the sequence! If we go back several levels higher, back to the full transformer model, we can see what we learned about the decoder part of the full transformer model. It is what we call, auto-regressive: it outputs values that are then used as its input values. We repeat this operations as we like. It is based off of the masked self-attention layer, which allows to have word embeddings which have access to the context on the left side of the word. If you look at the diagram however, you'll see that we haven't seen one of the aspects of the decoder. That is: cross-attention. There is a second aspect we haven't seen, which is it's ability to convert features to words; heavily linked to the cross attention mechanism. However, these only apply in the "encoder-decoder" transformer, or the "sequence-to-sequence" transformer (which can generally be used interchangeably). We recommend you check out the video on encoder-decoders to get an idea of how the decoder can be used as a component of a larger architecture!