Bert, NLP.
its individual characters. Here are some examples of the tokens contained in the vocabulary. list(tokenizer.vocab.keys())[5000:5020] ['knight', 'lap', 'survey', 'ma', '##ow', 'noise', 'billy', '##ium', 'shooting', 'guide', 'bedroom', 'priest', 'resistance', 'motor', 'homes', 'sounded', 'giant',