Em Algorithm, Machine Learning, Data Mining, Jensens Inequality, Bayes Theorem.
than or equal to zero with equality when the two distributions are equal, q(z) = p(z). It is not symmetric, KL(p||q) ≠ KL(q||p), so it is not a distance measure. It can be interpreted as the number of extra bits needed to encode the data using distribution q(z) when the true distribution is p(z)