Essential Math, Editors Pick, Mathematics, Data Science, Machine Learning.
Replacing with the expressions of the cross-entropy and the entropy, you get: The KL divergence is always non-negative. Since the entropy H(P) is identical to the cross-entropy H(P, P), and because the smallest cross-entropy is between identical distributions ( H(P, P)), H(P, Q) is necessarily