NLP, Data Science, Artificial Intelligence, Machine Learning, Language Model.
red fox”) = 1 / Pnorm(“a red fox.”) = 6 …which, as expected, is a higher perplexity than the one produced by the well-trained language model. Perplexity and Entropy Perplexity can be computed also starting from the concept of Shannon entropy. Let’s call H(W) the entropy of the language model when