Deep Learning, Artificial Intelligence, Naturallanguageprocessing, NLP, Machine Learning.
For now, I think this is a solid tutorial on the basics of building token to index lists. Colab Notebook: https://github.com/CShorten/CORD-19-Mining/blob/main/Tokenization_Tutorial.ipynb NLTK Tokenizer The NLTK tokenizer is pretty naive compared to things like Byte-Pair encoding or Subword