Data Science, Machine Learning, NLP, Transformers, Word Embeddings.
tokens. print('Tokenized: ', tokenizer.tokenize(sentences[0])[:15]) # Print the sentence mapped to token ids. print('Token IDs: ', tokenizer.convert_tokens_to_ids(tokenizer.tokenize(sentences[0]))[:15]) Note that some tokens may look like this: [‘aa’, ‘##th’, ‘##ur’, ‘pen’, ‘##dra’, ‘##gon’]. This