In this video we will see how you can create your own tokenizer from scratch! To create your own tokenizer you will have to think about each of the operations involved in tokenization, namely: normalization, pre-tokenization, model, post-processing and decoding. If you don't know what normalization, pre-tokenization and the model are, I advise you to go and see the videos linked below. The post processing gathers all the modifications that we will carry out on the tokenized text. It can include the addition of special tokens, the creation of an attention mask but also the generation of a list of token ids. The decoding operation occurs at the very end and will allow passing from the sequence of ids in a sentence. For example, in our example, we can see that the hashtags have been removed and the tokens composing the word "today" have been grouped together. In a fast tokenizer, all these components are gathered in the backend_tokenizer attribute. As you can see with this small code snippet, it is an instance of a tokenizer from the tokenizers library. So, to create your own transformers tokenizer you will have to follow these steps: first create a training dataset; second create and train a tokenizer with the tokenizers library and third load this tokenizer into transformers tokenizer. To understand these steps, I propose that we recreate a BERT tokenizer. The first thing to do is to create a dataset. With this code snippet you can create an iterator on the dataset wikitext-2-raw-v1 which is a rather small dataset in English. We attack here the big part: the design of our tokenizer with the tokenizers library. We start by initializing a tokenizer instance with a WordPiece model because it is the model used by BERT. Then we can define our normalizer. We will define it as a succession of 2 normalizations used to clean up characters not visible in the text, 1 lowercasing normalization and 2 normalizations used to remove accents. For the pre-tokenization, we will chain two pre_tokenizer. The first one separating the text at the level of spaces and the second one isolating the punctuation marks. Now, we can define the trainer that will allow us to train the WordPiece model chosen at the beginning. To carry out the training, we will have to choose a vocabulary size, here we choose twenty-five thousand and also announce the special tokens that we absolutely want to add to our vocabulary. In one line of code, we can train our WordPiece model using the iterator we defined earlier. Once the model has been trained, we can retrieve the ids of the special class and separation tokens because we will need them to post-process our sequence. Thanks to the TemplateProcessing class, we can add the CLS token at the beginning of each sequence and the SEP token at the end of the sequence and between two sentences if we tokenize a text pair. Finally, we just have to define our decoder which will allow us to remove the hashtags at the beginning of the tokens that must be reattached to the previous token. And there it ist, you have all the necessary lines of code to define your own tokenizer. Now that we have a brand new tokenizer with the tokenizers library we just have to load it into a fast tokenizer from the transformers library. Here again we have several possibilities. We can load it in the generic class "PreTrainedTokenizerFast" or in the BertTokenizerFast class since we have built a bert type tokenizer here. I hope this video has helped you understand how you can create your own tokenizer and that you are ready to navigate the tokenizers library documentation to choose the components for your brand-new tokenizer!