[[open-in-colab]]
In this tutorial, we'll explore how to preprocess your data using 🤗 Transformers. The main tool for this is what we
call a tokenizer. You can build one using the tokenizer class associated to the model
you would like to use, or directly with the [AutoTokenizer] class.
As we saw in the quick tour, the tokenizer will first split a given text in words (or part of
words, punctuation symbols, etc.) usually called tokens. Then it will convert those tokens into numbers, to be able
to build a tensor out of them and feed them to the model. It will also add any additional inputs the model might expect
to work properly.
If you plan on using a pretrained model, it's important to use the associated pretrained tokenizer: it will split
the text you give it in tokens the same way for the pretraining corpus, and it will use the same correspondence
