Let's take a look at word-based tokenization. Word-based tokenization is the idea of splitting the raw text into words, by splitting on spaces or other specific rules like punctuation. In this algorithm, each word has a specific number, an "ID", attributed to it. In this example, "Let's" has the ID 250, do has ID 861, and tokenization followed by an exclamation point has the ID 345. This approach is interesting, as the model has representations that are based on entire words. The information held in a single number is high as a word contains a lot of contextual and semantic information in a sentence. However, this approach does have its limits. For example, the word dog and the word dogs are very similar, and their meaning is close. However, the word-based tokenization will attribute entirely different IDs to these two words, and the model will therefore learn different meanings for these two words. This is unfortunate, as we would like the model to understand that these words are indeed related and that dogs is the plural form of the word dog. Another issue with this approach is that there are a lot of different words in a language. If we want our model to understand all possible sentences in that language, then we will need to have an ID for each different word, and the total number of words, which is also known as the vocabulary size, can quickly become very large. This is an issue because each ID is mapped to a large vector that represents the word's meaning, and keeping track of these mappings requires an enormous number of weights when the vocabulary size is large. If we want our models to stay lean, we can opt for our tokenizer to ignore certain words that we don't necessarily need. For example, when training our tokenizer on a text, we might want to take the 10,000 most frequent words in that text to create our basic vocabulary, instead of taking all of that language's words. The tokenizer will know how to convert those 10,000 words into numbers, but any other word will be converted to the out-of-vocabulary word, or the "unknown" word. This can rapidly become an issue: the model will have the exact same representation for all words that it doesn't know, which will result in a lot of lost information.