[[open-in-colab]]
On this page, we will have a closer look at tokenization.
As we saw in the preprocessing tutorial, tokenizing a text is splitting it into words or
subwords, which then are converted to ids through a look-up table. Converting words or subwords to ids is
straightforward, so in this summary, we will focus on splitting a text into words or subwords (i.e. tokenizing a text).
More specifically, we will look at the three main types of tokenizers used in 🤗 Transformers: Byte-Pair Encoding
(BPE), WordPiece, and SentencePiece, and show examples
of which tokenizer type is used by which model.
Note that on each model page, you can look at the documentation of the associated tokenizer to know which tokenizer
type was used by the pretrained model. For instance, if we look at [BertTokenizer], we can see
