Answers.Body: <p>In the context of Variational Inference (VI): the KL allows you to move from the unknown posterior <span class="math-container">$p(z \mid x)$</span>, to the known joint <span class="math-container">$p(z,x)=p(x|z)p(z)$</span> and optimize only the ELBO. You cannot do this with L2.</p>
  <p><span class="math-container">$p(z|x)$</span> is the desired posterior, of which you cannot calculate the evidence (i.e., using Bayes formula we can set: <span class="math-container">$p(z|x) = \frac{p(x|z)p(z)}{\int_z p(x|z)p(z)dz}$</span>, and you can't calculate the integral in the denominator [also denoted by <span class="math-container">$p(x)$</span>] due to it's intractability).</p>
  <p>Now suppose <span class="math-container">$q$</span> is a Variational distribution (e.g. a family of Gaussians which you can control), VI tries to approximates <span class="math-container">$p(z|x)$</span> by <span class="math-container">$q$</span> by minimizing their KL divergence.</p>
  <p><span class="math-container">$$KL(q(z)||p(z|x)) = \int_z q(z) \log \frac{q(z)}{p(z|x)}dz = \mathbb E_q[\log q(z)]-\mathbb E_q[\log \frac{p(x|z)p(z)}{p(x)}] =$$</span></p>
  <p><span class="math-container">$$ -\mathbb E_q[\log p(x|z)p(z)] + \mathbb E_q[\log q(z)] + \mathbb E_q[\log p(x)] = -ELBO(q) + \log p(x)
  $$</span></p>
  <p>Since you're only optimizing <span class="math-container">$q$</span> (it's the only thing you can control), you can discard the unknown and difficult to compute normalizing constant <span class="math-container">$p(x)$</span>.</p>
  <p>If you would use the (squared) L2 norm you would get:
  <span class="math-container">$$\int_z [q(z)-p(z|x)]^2dz = \int_z [q^2(z)-2q(z)p(z|x)+p^2(z|x)]dz $$</span>
  While the 3rd term doesn't depend on q, the 2nd term does, and it also requires <span class="math-container">$p(x)$</span> to compute.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T08:17:23.713Z
Answers.Text: Maybe you should clarify that $q$ is the variational distribution (e.g. a family of Gaussians), i.e. the distribution that you're using to approximate the unknown one that you want to learn. From this answer, it's not fully clear why the KL divergence is really better than the L2 norm, which you seem to have defined as an expectation with respect to $z$. I recommend that you edit your post in order to clarify it.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T10:49:10.560Z
Answers.Text: I added a crash course in VI, but some knowledge is needed of VI in order to understand this. As implied by the "In the context of Variational Inference" that starts the answer.
Answers.UserDisplayName: Maverick Meerkat
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T05:58:52.473Z
Answers.LastActivityDate: 2022-05-30T10:46:00.427Z
Answers.LastEditDate: 2022-05-30T10:46:00.427Z
Answers.UserDisplayName: Maverick Meerkat
Answers.Body: <p>KL-divergence is a measure on probability distributions. It essentially captures the information loss between ground truth distribution and predicted.</p>
  <p>L2-norm/MSE/RMSE doesn't do well with probabilities, because of the power operations involved in the calculation of loss. Probabilities, being fractions under 1, are significantly affected by any power operations (square or root), and considering we are calculating the squares of differences of probabilities, the values that are summed are abnormally small, essentially barely learning anything as the random initialization itself starts with an abnormally small loss, almost always staying constant.</p>
  <p>L1 norm, on the other hand, does not have any power operations, making it relatively acceptable.</p>
  <p>Loss functions, such as Kullback-Leibler-divergence or Jensen-Shannon-Divergence, are preferred for probability distributions because of the statistical meaning they hold. KL-Divergence, as mentioned before, is a statistical measure of information loss between distributions, or, in other words, assuming <span class="math-container">$Q$</span> is the ground truth distribution, KL-Divergence is a measure of how much <span class="math-container">$P$</span> deviates from <span class="math-container">$Q$</span>. Also, considering probability distributions, convergence is much stronger in measures of Information Loss such KL-Divergence.</p>
  <p>More clarity on the motivation behind Kullback-Leibler can be read <a href="https://math.stackexchange.com/q/90537">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-16T09:44:09.160Z
Answers.LastActivityDate: 2022-05-30T08:24:02.723Z
Answers.LastEditDate: 2022-05-30T08:24:02.723Z
Answers.UserDisplayName: ashenoy
Body: <p>I've recently encountered different articles that are recommending to use the KL divergence instead of the MSE/RMSE (as the loss function), when trying to learn a probability distribution, but none of the articles are giving a clear reasoning why the first is better than the others.</p>
  <p>Could anyone give me a strong argument why the KL divergence is suitable for this?</p>
  
Tags: <comparison><objective-functions><optimization><mean-squared-error><kl-divergence>
Title: What are the advantages of the Kullback-Leibler over the MSE/RMSE?
UserDisplayName: razvanc92
ThreadText: # What are the advantages of the Kullback-Leibler over the MSE/RMSE?
  
  (Asked by: username_0 on 2019-09-11)
  
  I've recently encountered different articles that are recommending to use the KL divergence instead of the MSE/RMSE (as the loss function), when trying to learn a probability distribution, but none of the articles are giving a clear reasoning why the first is better than the others.
  
  Could anyone give me a strong argument why the KL divergence is suitable for this?
  
  **Tags:** comparison, objective-functions, optimization, mean-squared-error, kl-divergence
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-05-30. Score: 1 | Accepted answer
  
  In the context of Variational Inference (VI): the KL allows you to move from the unknown posterior $p(z \mid x)$, to the known joint $p(z,x)=p(x|z)p(z)$ and optimize only the ELBO. You cannot do this with L2.
  
  $p(z|x)$ is the desired posterior, of which you cannot calculate the evidence (i.e., using Bayes formula we can set: $p(z|x) = \frac{p(x|z)p(z)}{\int\_z p(x|z)p(z)dz}$, and you can't calculate the integral in the denominator [also denoted by $p(x)$] due to it's intractability).
  
  Now suppose $q$ is a Variational distribution (e.g. a family of Gaussians which you can control), VI tries to approximates $p(z|x)$ by $q$ by minimizing their KL divergence.
  
  $$KL(q(z)||p(z|x)) = \int\_z q(z) \log \frac{q(z)}{p(z|x)}dz = \mathbb E\_q[\log q(z)]-\mathbb E\_q[\log \frac{p(x|z)p(z)}{p(x)}] =$$
  
  $$ -\mathbb E\_q[\log p(x|z)p(z)] + \mathbb E\_q[\log q(z)] + \mathbb E\_q[\log p(x)] = -ELBO(q) + \log p(x)
  $$
  
  Since you're only optimizing $q$ (it's the only thing you can control), you can discard the unknown and difficult to compute normalizing constant $p(x)$.
  
  If you would use the (squared) L2 norm you would get:
  $$\int\_z [q(z)-p(z|x)]^2dz = \int\_z [q^2(z)-2q(z)p(z|x)+p^2(z|x)]dz $$
  While the 3rd term doesn't depend on q, the 2nd term does, and it also requires $p(x)$ to compute.
  
  > **Comments:**
  > **username_2**: Maybe you should clarify that $q$ is the variational distribution (e.g. a family of Gaussians), i.e. the distribution that you're using to approximate the unknown one that you want to learn. From this answer, it's not fully clear why the KL divergence is really better than the L2 norm, which you seem to have defined as an expectation with respect to $z$. I recommend that you edit your post in order to clarify it.  
  > **username_1**: I added a crash course in VI, but some knowledge is needed of VI in order to understand this. As implied by the "In the context of Variational Inference" that starts the answer.  
  
  ---
  
  ### Answer by username_3 on 2019-09-16. Score: 2
  
  KL-divergence is a measure on probability distributions. It essentially captures the information loss between ground truth distribution and predicted.
  
  L2-norm/MSE/RMSE doesn't do well with probabilities, because of the power operations involved in the calculation of loss. Probabilities, being fractions under 1, are significantly affected by any power operations (square or root), and considering we are calculating the squares of differences of probabilities, the values that are summed are abnormally small, essentially barely learning anything as the random initialization itself starts with an abnormally small loss, almost always staying constant.
  
  L1 norm, on the other hand, does not have any power operations, making it relatively acceptable.
  
  Loss functions, such as Kullback-Leibler-divergence or Jensen-Shannon-Divergence, are preferred for probability distributions because of the statistical meaning they hold. KL-Divergence, as mentioned before, is a statistical measure of information loss between distributions, or, in other words, assuming $Q$ is the ground truth distribution, KL-Divergence is a measure of how much $P$ deviates from $Q$. Also, considering probability distributions, convergence is much stronger in measures of Information Loss such KL-Divergence.
  
  More clarity on the motivation behind Kullback-Leibler can be read [here](https://math.stackexchange.com/q/90537).
  
  
  ---
  
  