to new words (as long as those new words do not include symbols that were not in the base vocabulary). For instance,
the word "bug" would be tokenized to ["b", "ug"] but "mug" would be tokenized as ["<unk>", "ug"] since
the symbol "m" is not in the base vocabulary. In general, single letters such as "m" are not replaced by the
"<unk>" symbol because the training data usually includes at least one occurrence of each letter, but it is likely
to happen for very special characters like emojis.
As mentioned earlier, the vocabulary size, i.e. the base vocabulary size + the number of merges, is a hyperparameter
to choose. For instance GPT has a vocabulary size of 40,478 since they have 478 base characters
and chose to stop training after 40,000 merges.
Byte-level BPE
A base vocabulary that includes all possible base characters can be quite large if e.g. all unicode characters are
