Answers.Body: <p>You can know it better, if you know the concept of <a href="https://en.wikipedia.org/wiki/Entropy_(information_theory)" rel="nofollow noreferrer">entropy</a>:</p>
  <blockquote>
  <p>Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also called the surprisal) of an event <span class="math-container">${\displaystyle E}$</span>  is an increasing function of the reciprocal of the <span class="math-container">${\displaystyle p(E)}$</span> of the event, precisely <span class="math-container">${\displaystyle I(E)=-\log _{2}(p(E))=\log _{2}(1/p(E))}$</span>. Shannon defined the entropy Η of a discrete random variable <span class="math-container">${\textstyle X}$</span> with possible values <span class="math-container">${\textstyle \left\{x_{1},\ldots ,x_{n}\right\}}$</span> and probability mass function <span class="math-container">${\textstyle \mathrm {P} (X)}$</span> as:
  <span class="math-container">$${\displaystyle \mathrm {H} (X)=\operatorname {E} [\operatorname {I} (X)]=\operatorname {E} [-\log(\mathrm {P} (X))].}$$</span>
  Here <span class="math-container">${\displaystyle \operatorname {E} }$</span> is the expected value operator, and <span class="math-container">$I$</span> is the information content of <span class="math-container">${\displaystyle I(X)}$</span> is itself a random variable. The entropy can explicitly be written as
  <span class="math-container">$${\displaystyle \mathrm {H} (X)=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{b}\mathrm {P} (x_{i})}}$$</span>
  where <span class="math-container">$b$</span> is the base of the logarithm used.</p>
  </blockquote>
  <p>Now, the KL divergence, try to find the cross entropy of two probability distributions.</p>
  <p>So, what is the <a href="https://en.wikipedia.org/wiki/Cross_entropy" rel="nofollow noreferrer">cross entropy</a>:</p>
  <blockquote>
  <p>The cross entropy between two probability distributions <span class="math-container">${\displaystyle p}$</span> and <span class="math-container">${\displaystyle q}$</span> over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution <span class="math-container">${\displaystyle q}$</span>, rather than the true distribution  <span class="math-container">${\displaystyle p}$</span>.</p>
  <p>The cross entropy for the distributions <span class="math-container">${\displaystyle p}$</span> and  <span class="math-container">${\displaystyle q}$</span> over a given set is defined as follows:
  <span class="math-container">$${\displaystyle H(p,q)=\operatorname {E} _{p}[-\log q]}.$$</span>
  The definition may be formulated using the Kullback–Leibler divergence<span class="math-container">${\displaystyle D_{\mathrm {KL} }(p\|q)}$</span> of  <span class="math-container">${\displaystyle q}$</span> from <span class="math-container">${\displaystyle p}$</span> (also known as the relative entropy of <span class="math-container">${\displaystyle p}$</span> with respect to <span class="math-container">${\displaystyle q}$</span>).
  <span class="math-container">$${\displaystyle H(p,q)=H(p)+D_{\mathrm {KL} }(p\|q)},$$</span>
  where <span class="math-container">${\displaystyle H(p)}$</span> is the entropy of <span class="math-container">${\displaystyle p}$</span>.</p>
  </blockquote>
  <p>Now, as we want to approximate a distribution function <span class="math-container">$p$</span> with other distribution function <span class="math-container">$q$</span>, we want to minimize the cross entropy between these two. The first part <span class="math-container">$H(p)$</span> could not be changed as we want to find a distribution <span class="math-container">$q$</span> and <span class="math-container">$p$</span> is given. Hence, we need to minimize <span class="math-container">$KL$</span> divergence of these two, to minimize the cross entropy and better approximation for the <span class="math-container">$p$</span> distribution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T08:10:36.950Z
Answers.LastActivityDate: 2019-10-22T08:18:37.773Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: OmG
Body: <p>I'm reading about <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence" rel="nofollow noreferrer">the KL divergence on Wikipedia</a>. I don't understand how the equation gives &quot;information gained&quot; as it says in the &quot;Interpretations&quot; section</p>
  <blockquote>
  <p>Expressed in the language of Bayesian inference, <span class="math-container">${\displaystyle D_{\text{KL}}(P\parallel Q)}$</span> is a measure of the <strong>information gained</strong> by revising one's beliefs from the prior probability distribution <span class="math-container">$Q$</span> to the posterior probability distribution <span class="math-container">$P$</span></p>
  </blockquote>
  <p>I was under the impression that KL divergence is a way of measure the difference between distributions (used in autoencoders to determine the difference between the input and the output generated from latents).</p>
  <p>How does the equation <span class="math-container">$$D_{KL}(P \| Q)=\sum P(x) \log \left( \frac{P(X)}{Q(X)} \right)$$</span> give us a divergence? Also, in encoding and decoding algorithms that use KL divergence, is the goal to minimize <span class="math-container">$D_{KL}(P \| Q)$</span>?</p>
  
Tags: <machine-learning><terminology><variational-autoencoder><information-theory><kl-divergence>
Title: How does the Kullback-Leibler divergence give "knowledge gained"?
UserDisplayName: Gooby
ThreadText: # How does the Kullback-Leibler divergence give "knowledge gained"?
  
  (Asked by: username_0 on 2019-10-22)
  
  I'm reading about [the KL divergence on Wikipedia](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence). I don't understand how the equation gives "information gained" as it says in the "Interpretations" section
  
  > Expressed in the language of Bayesian inference, ${\displaystyle D\_{\text{KL}}(P\parallel Q)}$ is a measure of the **information gained** by revising one's beliefs from the prior probability distribution $Q$ to the posterior probability distribution $P$
  
  I was under the impression that KL divergence is a way of measure the difference between distributions (used in autoencoders to determine the difference between the input and the output generated from latents).
  
  How does the equation $$D\_{KL}(P \| Q)=\sum P(x) \log \left( \frac{P(X)}{Q(X)} \right)$$ give us a divergence? Also, in encoding and decoding algorithms that use KL divergence, is the goal to minimize $D\_{KL}(P \| Q)$?
  
  **Tags:** machine-learning, terminology, variational-autoencoder, information-theory, kl-divergence
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-10-22. Score: 3 | Accepted answer
  
  You can know it better, if you know the concept of [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)):
  
  > Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also called the surprisal) of an event ${\displaystyle E}$ is an increasing function of the reciprocal of the ${\displaystyle p(E)}$ of the event, precisely ${\displaystyle I(E)=-\log \_{2}(p(E))=\log \_{2}(1/p(E))}$. Shannon defined the entropy Η of a discrete random variable ${\textstyle X}$ with possible values ${\textstyle \left\{x\_{1},\ldots ,x\_{n}\right\}}$ and probability mass function ${\textstyle \mathrm {P} (X)}$ as:
  > $${\displaystyle \mathrm {H} (X)=\operatorname {E} [\operatorname {I} (X)]=\operatorname {E} [-\log(\mathrm {P} (X))].}$$
  > Here ${\displaystyle \operatorname {E} }$ is the expected value operator, and $I$ is the information content of ${\displaystyle I(X)}$ is itself a random variable. The entropy can explicitly be written as
  > $${\displaystyle \mathrm {H} (X)=-\sum \_{i=1}^{n}{\mathrm {P} (x\_{i})\log \_{b}\mathrm {P} (x\_{i})}}$$
  > where $b$ is the base of the logarithm used.
  
  Now, the KL divergence, try to find the cross entropy of two probability distributions.
  
  So, what is the [cross entropy](https://en.wikipedia.org/wiki/Cross_entropy):
  
  > The cross entropy between two probability distributions ${\displaystyle p}$ and ${\displaystyle q}$ over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution ${\displaystyle q}$, rather than the true distribution ${\displaystyle p}$.
  >
  > The cross entropy for the distributions ${\displaystyle p}$ and ${\displaystyle q}$ over a given set is defined as follows:
  > $${\displaystyle H(p,q)=\operatorname {E} \_{p}[-\log q]}.$$
  > The definition may be formulated using the Kullback–Leibler divergence${\displaystyle D\_{\mathrm {KL} }(p\|q)}$ of ${\displaystyle q}$ from ${\displaystyle p}$ (also known as the relative entropy of ${\displaystyle p}$ with respect to ${\displaystyle q}$).
  > $${\displaystyle H(p,q)=H(p)+D\_{\mathrm {KL} }(p\|q)},$$
  > where ${\displaystyle H(p)}$ is the entropy of ${\displaystyle p}$.
  
  Now, as we want to approximate a distribution function $p$ with other distribution function $q$, we want to minimize the cross entropy between these two. The first part $H(p)$ could not be changed as we want to find a distribution $q$ and $p$ is given. Hence, we need to minimize $KL$ divergence of these two, to minimize the cross entropy and better approximation for the $p$ distribution.
  
  
  ---
  
  