Machine Learning, NLP, Information Technology, Data, AI.
useful in many cases. For example, it constitutes the penalty to pay in code length when trying to encode a source as if it was distributed according to . One of the reasons the KL divergence is so useful is the property , with equality if and only if . Note however, that , which is why the KL