considered as base characters. To have a better base vocabulary, GPT-2 uses bytes
as the base vocabulary, which is a clever trick to force the base vocabulary to be of size 256 while ensuring that
every base character is included in the vocabulary. With some additional rules to deal with punctuation, the GPT2's
tokenizer can tokenize every text without the need for the  symbol. GPT-2 has a vocabulary
size of 50,257, which corresponds to the 256 bytes base tokens, a special end-of-text token and the symbols learned
with 50,000 merges.
WordPiece
WordPiece is the subword tokenization algorithm used for BERT, DistilBERT, and Electra. The algorithm was outlined in Japanese and Korean
Voice Search (Schuster et al., 2012) and is very similar to
BPE. WordPiece first initializes the vocabulary to include every character present in the training data and
