The fast tokenizers of the Transformers library are fast, but they also implement features that will be super useful for data pre-processing and post-processing. Let's have a look at them! First, let's have a look at the usual output of a tokenizer. We get input IDs that correspond to tokens, but we lose a lot of information in the process. For instance, here the tokenization is the same for the two sentences, even if one has several more spaces than the other. Just having the input IDs is thus not enough if we want to match some tokens with a span of text (something we will need to do when tackling question answering for instance). It's also difficult to know when two tokens belong to the same word or not: it looks easy when you just look at the output of a BERT tokenizer, we just need to look for the ##. But other tokenizers have different ways to tokenize parts of words. For instance RoBERTa adds this special G symbol to mark the tokens at the beginning of a word, and T5 uses this special underscore symbol for the same purpose. Thankfully, the fast tokenizers keep track of the word each token comes from, with a word_ids method you can use on their outputs. The output is not necessarily clear, but assembled together in a nice table like this, we can look at the word position for each token. Even better, the fast tokenizers keep track of the span of characters each token comes from, and we can get them when calling it on one (or several) text by adding the return_offsets_mapping=True argument. In this instance, we can see how we jump positions between the ##s token and the super token, because of the multiple spaces in the initial sentence. To enable this, the fast tokenizers store additional information at each step of their internal pipeline. That internal pipeline consists of normalization, where we apply some cleaning to the text, like lowercasing or removing the accents;() pre-tokenization, which is where we split the texts into words;() then we apply the model of the tokenizer, which is where the words are splits into tokens,() before finally doing the post-processing, where special tokens are added. From the beginning to the end of the pipeline, the tokenizer keeps track of each span of text that corresponds to each word, then each token. We will see how useful it is when we tackle the following tasks: when doing masked language modeling, one variation that gets state-of-the-art results is to mask all the tokens of a given word instead of randomly chosen tokens. This will require us to use the word IDs we saw. When doing token classification, we'll need to convert the labels we have on words, to labels on each tokens. As for the offset mappings, it will be super useful when we need to convert token positions in a sentence into a span of text, which we will need to know when looking at question answering or when grouping the tokens corresponding to the same entity in token classification. To have a look at these tasks, check the videos linked below!