Machine Learning, Markov Chains, Statistics, NLP, Fun.
Moving on, here’s how we initialize our “word after k-sequence” counts matrix for an arbitrary k (in this case, 2). There are 2185918 words in the corpus, and 429582 different sequences of 2 words, each followed by one of 32663 words. That means only slightly over 0.015% of our matrix’s components