Essential Math, Editors Pick, Mathematics, Data Science, Machine Learning.
compared to the true distribution P(x). It tells you how different the two distributions are. Mathematically, the KL divergence between two distributions P(x) and Q(x), denoted as D_KL(P||Q), is expressed as the difference between the cross-entropy of P(x) and Q(x) and the entropy of P(x):