Bert, NLP.
tokenizer. tokenized_text = tokenizer.tokenize(marked_text) # Print out the tokens. print (tokenized_text) ['[CLS]', 'here', 'is', 'the', 'sentence', 'i', 'want', 'em', '##bed', '##ding', '##s', 'for', '.', '[SEP]'] Notice how the word “embeddings” is represented: ['em', '##bed', '##ding', '##s']