Why KL divergence close to zero when Q close to P? I was understanding cross-entropy and ended up understanding KL divergence. I learnt Cross entropy is Entropy + KL Divergence:

H(P, Q) = H(P) + D_KL(P||Q)

Minimizing Cross-entropy means minimizing KL Divergence. I further read that minimizing KL divergence means we are trying to make Q close to P. But, I really wanted to know why this happens? I read from many sources that when Q close to P, DKL close to zero but I didn't find any proper justification for this. I wonder if somebody has better insights on this.

For discrete random variables $P$ and $Q$, the KL-divergence is defined as
$$
D_{KL}(P || Q) = \sum_x P(x) \ln\frac{P(x)}{Q(x)}
$$
So, as $Q \rightarrow P$, the ratio $P(x)/Q(x)$ approaches $1$ for all $x$ and the logarithm $\ln P(x)/Q(x)$ approaches zero. As probabilities are bounded to the range $[0, 1]$, each term in the sum, $P(x) \ln\frac{P(x)}{Q(x)}$ also approaches zero and, consequently, the whole sum also approaches zero.
So far the mathematical formalism. For some intuition behind it, you may consult my answer here.
