In this video we take a look at the mysterious sounding metric called Perplexity. You might have encountered perplexity when reading about generative models. You can see two examples here from the original transformer paper “Attention is all you need” as well as the more recent GPT-2 paper. Perplexity is a common metric to measure the performance of language models. The smaller the value the better the performance. But what does it actually mean and how can we calculate it? A very common quantity in machine learning is the likelihood. We can calculate the likelihood as the product of each token’s probability What this means is that for each token we use the language model to predict its probability based on the previous tokens. In the end we multiply all probabilities to get the Likelihood. With the likelihood we can calculate another important quantity: the cross entropy. You might already have heard about cross-entropy when looking at loss function. Cross-entropy is often used as a loss function in classification. In language modeling we predict the next token which also is  a classification task. Therefore, if we want to calculate the cross entropy of an example we can simply pass it to the model with the inputs as labels. The loss return by the model then corresponds the cross entropy. We are now only a single operation away from calculating the perplexity. By exponentiating the cross-entropy we get the perplexity. So you see that the perplexity is closely related to the loss. Keep in mind that the loss is only a weak proxy for a model’s ability to generate quality text and the same is true for perplexity. For this reason one usually also calculates more sophisticated metrics such as BLEU or ROUGE on generative tasks.