Machine Learning, Markov Chains, Statistics, NLP, Fun.
this experiment, I decided to treat anything between two spaces as a word or token. Conventionally, in NLP we treat punctuation marks (like ‘,’ or ‘.’) as tokens as well. To solve this, I will first add padding in the form of two spaces to every punctuation mark. Here’s the code for that small