The tokenization pipeline involves several steps that convert raw text into numbers. In this video, we will see what happens during the pre-tokenization step. The pre-tokenization operation is the operation performed after the normalization of the text and before the application of the tokenization algorithm. This step consists in applying rules that do not need to be learned to perform a first division of the text. Let's look at how several tokenizers pre_tokenize this example. The gpt 2 pretokenization divides the text on spaces and some punctuation - but the apostrophe is not a division criterion for example. We also notice that spaces have been replaced by a capital G with a dot above. Albert's pre-tokenization divides the text at the level of spaces, adds a space at the beginning of the sentence and replaces spaces with a special underscore. Finally, Bert's pre-tokenization divides the text at the level of punctuation and spaces. Unlike the previous tokenizers, spaces are not transformed and integrated to the tokens produced with this pre-tokenizer. Through these 3 examples, we could observe the two main types of operations brought by the pre-tokenization: some changes on the text and the division of the string into tokens that can be associated to words. Finally, the "backend_tokenizer" of the fast tokenizers also allows to test the pre-tokenization operation very easily thanks to its "pre_tokenize_str" method. We notice that the output of this operation is composed of both tokens and offsets which allow to link the token to its position in the text given in input of the method. This operation defines the largest tokens that can be produced by the tokenization or in other words the barriers of the sub-tokens which will be produced then.