Let's see how we can preprocess our data for masked language modeling. As a reminder, masked language modeling is when a model needs to fill the blanks in a sentence. To do this, you just need texts, no labels, as this is a self-supervised problem. To apply this on your own data, just make sure you have all your texts gathered in one column of your dataset. Before we start randomly masking things, we will need to somehow make all those texts the same length to batch them together. The first way to make all the texts the same length is the one we used in text classification. let's pad the short texts and truncate the long ones. As we have seen when we processed data for text classification, this is all done by our tokenizer with the right options for padding and truncation. This will however make us lose a lot of texts if the examples in our dataset are very long, compared to the context length we picked. Here, all the portion in gray is lost. This is why a second way to generate samples of text with the same length is to chunk our text in pieces of context lengths, instead of discarding everything after the first chunk. There will probably be a remainder of length smaller than the context size, which we can choose to keep and pad or ignore. Here is how we can apply this in practice, by just adding the return overflowing tokens option in our tokenizer call. Note how this gives us a bigger dataset! This second way of chunking is ideal if all your texts are very long, but it won't work as nicely if you have a variety of lengths in the texts. In this case, the best option is to concatenate all your tokenized texts in one big stream, with a special tokens to indicate when you pass from one document to the other, and only then split the big stream into chunks. Here is how it can be done with code, with one loop to concatenate all the texts and another one to chunk it. Notice how it reduces the number of samples in our dataset here, there must have been quite a few short entries! Once this is done, the masking is the easy part. There is a data collator designed specifically for this in the Transformers library. You can use it directly in the Trainer, or when converting your datasets to tensorflow datasets before doing Keras.fit, with the to_tf_dataset method.