KL Divergence


## Relative Entropy, Kullback Leibler Divergence, **I-divergence**
a metric to compare two distributions
두 분포가 같다면 0, 그 이외의 경우에는 0보다 크다 (즉 기대한 정보와 결과의 차이 prior, posterior)
divergence means just difference
bigger than 0 due to the [Gibbs' inequality](https://texonom.com/gibbs-inequality-924d4f81cbb345e4a3c4436d64e29abe)
KL-divergence를 minimize하는 것 또한 결국 log likelihood를 maximize하는 것과 같다

### Popular Distance
Cross-entropy에서 entropy를 뺀 값. 거리 개념이 아니다
In case of KL divergence, we have the correspondence between MLE and KL matching
![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F69115b1b-4eec-456c-b839-317d864089a9%2FUntitled.png?table=block&id=9462b66e-5813-4f97-ab47-fed0f8302fd6&cache=v2)
has an analytic solution if both p and q follows the normal distribution



> [KL divergence - 공돌이의 수학정리노트 (Angelo's Math Notes)](https://angeloyeo.github.io/2020/10/27/KL_divergence.html)






