In this video we take a look at the data processing necessary to train causal language models. Causal Language Modeling is the task of predicting the next token based on the previous token. Another term for Causal Language Modeling is Autoregressive Modeling. In the example that you see here the next token could for example be NLP or machine learning. A popular example of a Causal Language Model is the GPT family of models. To train such models such as GPT-2 we usually start with a large corpus of text files. These files can webpages scraped from the internet such as the Common Crawl dataset or they can be Python files from GitHub like you can see here. As a first step we need to tokenize these files such that we can feed them through a model. Here we show the tokenized texts as bars of various length illustrating the different sequence lengths. Normally, the text files come in various sizes and which results in various sequence length of the tokenized texts. Transformer models have a limited context length and depending on the data source it is possible that the tokenized texts are much longer than this context length. In this case we could just truncate the sequence to the context length but this would mean that we loose everything after the context length. Using the return overflowing tokens flag in the we can use the tokenizer to create chunks with each one being the size of the context length. Sometimes it can happen that the last chunk is too short if there aren’t enough tokens to fill it. In this case we would like to remove it. With the return_length keyword we also get the length of each chunk from the tokenizer. This function shows all the steps necessary to prepare the dataset. First we tokenize the dataset with the flags I just mentioned. Then we go through each chunk and if its length matches the context length we add it to the inputs we return. We can apply this function to the whole dataset and we make sure to use batches and remove the existing columns. We need to remove columns because we can create multiple samples per text and the shapes in the dataset would not match. If the context length is of similar length as the files this approach doesn't so well anymore. In this example both sample 1 and 2 are shorter than the context size and would be discarded with the previous approach. In this case it is better to first tokenize each sample without truncation and then concatenate the tokenized samples with an end of string, or EOS for short, token in between. Finally we can chunk this long sequence with the context length and we don’t loose any sequences because they are too short. So far we have only talked about the inputs for causal language modeling but not the labels needed for supervised training. When we do causal language modeling we don’t require any extra labels for the input sequences as the input sequences themselves are the labels. In this example when we feed the token “Trans” to the next token we want the model to predict is “formers”. In the next step we feed “Trans” and “formers” to the model and the label is the token “are”. This pattern continues and as you can see the input sequence is the label just shifted by one. Since the model only makes a prediction after the first token, the first element of the input sequence, in this case “Trans”, is not used as a label. Similarly, we do not have a label for the last token in the sequence since there is no token after the sequence ends. Let’s have a look at what we need to do to create the labels for causal language modeling in code.If we want to calculate the loss on a batch we can just pass the input_ids as labels and all the shifting is handled in the model internally. And the dataset is also ready to be used directly in the Trainer or keras.fit if you are using TensorFlow. So you see there is no magic involved in processing data for causal language modeling and only requires a few simple steps!