Machine Learning, Artificial Intelligence, Data Science.
and KL(p, q) ≠KL(q, p) (non-symmetry) KL(q, p) is called the reverse KL-divergence. The non-symmetry property of KL-divergence has some very significant implications. For example, let’s say the ground truth p is a bimodal distribution (the blue curve below) and we want to model it with a