that the model uses WordPiece.
Introduction
Splitting a text into smaller chunks is a task that is harder than it looks, and there are multiple ways of doing so.
For instance, let's look at the sentence "Don't you love 🤗 Transformers? We sure do."
A simple way of tokenizing this text is to split it by spaces, which would give:
["Don't", "you", "love", "🤗", "Transformers?", "We", "sure", "do."]
This is a sensible first step, but if we look at the tokens "Transformers?" and "do.", we notice that the
punctuation is attached to the words "Transformer" and "do", which is suboptimal. We should take the
punctuation into account so that a model does not have to learn a different representation of a word and every possible
punctuation symbol that could follow it, which would explode the number of representations the model has to learn.
