Answers.Body: <p>The KL divergence has slightly different interpretations depending on the context. The <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Interpretations" rel="nofollow noreferrer">related Wikipedia article</a> contains a section dedicated to these interpretations. Independently of the interpretation, the KL divergence is always defined as a <em>specific</em> function of the <a href="https://en.wikipedia.org/wiki/Cross_entropy" rel="nofollow noreferrer">cross-entropy</a> (which you should be familiar with before attempting to understand the KL divergence) between two distributions (in this case, probability mass functions)</p>
  
  <p><span class="math-container">\begin{align}
  D_\text{KL}(P\parallel Q) 
  &amp;= -\sum_{x\in\mathcal{X}} p(x) \log q(x) + \sum_{x\in\mathcal{X}} p(x) \log p(x) \\
   &amp;= H(P, Q) - H(P)
  \end{align}</span>
  where <span class="math-container">$H(P, Q)$</span> is the cross-entropy of the distribution <span class="math-container">$P$</span> and <span class="math-container">$Q$</span> and <span class="math-container">$H(P) = H(P, P)$</span>.</p>
  
  <p>The KL is not a metric, given that it does not obey the triangle inequality. In other words, in general, <span class="math-container">$D_\text{KL}(P\parallel Q) \neq D_\text{KL}(Q\parallel P) $</span>.</p>
  
  <p><em>Given that a neural network is trained to output the mean (which can be a scalar or a vector) and the variance (which can be a scalar, a vector or a matrix), why don't we use a metric like the MSE to compare means and variances?</em> When you use the KL divergence, you don't want to compare just numbers (or matrices), but probability distributions (more precisely, probability densities or mass functions), so you will not compare just the mean and the variance of two different distributions, but you will actually compare the distributions. See <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Basic_example" rel="nofollow noreferrer">the example of the application of the KL divergence</a> in the related Wikipedia article.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-31T02:41:47.583Z
Answers.Text: MSE doesn't compare variance, I know, but I could argue that my neural net is not a probabilistic function (it is a deterministic one), hence there is no variance. A natural gradient is then should be formed in MSE distance of the function output?
Answers.UserDisplayName: Phizaz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-30T18:39:37.330Z
Answers.LastActivityDate: 2019-10-30T18:52:26.117Z
Answers.LastEditDate: 2019-10-30T18:52:26.117Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Yes, Squared distances &amp; KL Divergence are not the same. Squared distance between means is not a useful metric as it doesn't gauge the amount of similarity between 2 distributions.</p>
  
  <p>When we compute <span class="math-container">\begin{align}
  D_\text{KL}(P\parallel Q) 
  \end{align}</span>
  We are computing the amount of information that is lost when we approximate <strong>P</strong> as <strong>Q</strong>. Ideally, we would want the KL divergence to be as low as possible.
  Here is an interesting article <a href="https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained" rel="nofollow noreferrer">https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained</a>
  where the author has explained KL Divergence with a toy example.</p>
  
  <p>I hope it helps :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-31T08:35:32.357Z
Answers.Text: It is true if the assumption is that the neural network is a probability function. However, the assumption is arbitrary. I could do equally well if I assume that the neural net is a deterministic function. The square distance makes much more sense now.
Answers.UserDisplayName: Phizaz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-30T19:43:00.560Z
Answers.LastActivityDate: 2019-10-30T19:43:00.560Z
Answers.UserDisplayName: Srivatsan Ramesh
Answers.Body: <p>I have been reading a lot about Natural Gradient and its use to find a descent direction. I found that <a href="https://wiseodd.github.io/techblog/2018/03/14/natural-gradient/" rel="nofollow noreferrer">this post</a> was the most clear.</p>
  <p>Consider a model <span class="math-container">$p$</span> parameterized by some parameters <span class="math-container">$\theta$</span> and we want to maximize the likelihood of observing our data <span class="math-container">$x$</span> under this model: <span class="math-container">$p(x|\theta)$</span>. To optimise this likelihood we can take steps in the the <strong>distribution space</strong>. Updating the parameters <span class="math-container">$\theta$</span> we need to measure how our likelihood changes and this is measure using the KLK divergence.</p>
  <p>Even though the KL divergence is not a &quot;proper&quot; distance metric as it is not symmetric, it is still quite informative about the similarity between distributions. It's practical because it can capture differences between distributions that the Euclidean metric (parameter-dependent) could not (see <a href="https://wiseodd.github.io/techblog/2018/03/14/natural-gradient/" rel="nofollow noreferrer">the same post</a> for a simple example).</p>
  <p>So answering your question is essentially answering which is the best between Natural Gradient Descent and &quot;Normal&quot; Gradient Descent in the Euclidean space where your loss is measured with a L2 norm.
  You can train the same model using both methods and you will just find different descent directions.</p>
  <p>Hopefully though, both will converge but in my opinion Natural Gradient descent should be superior in nature. It is just very expensive to actually compute because to find the direction in distribution space you need to compute the inverse Fisher matrix <span class="math-container">$F^{-1}$</span> or approximate it and that's quite costly as it is of size <span class="math-container">$n\times n$</span> where <span class="math-container">$n$</span> is the size of <span class="math-container">$\theta$</span> which is typically high in neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-26T09:55:10.123Z
Answers.LastActivityDate: 2020-06-26T09:55:10.123Z
Answers.UserDisplayName: maraboule
Body: <p>Natural gradient aims to do a steepest descent on the "function" space, a manifold that is independent from how the function is parameterized. It argues that the steepest descent on this function space is not the same as steepest descent on the parameter space. We should favor the former. </p>
  
  <p>Since, for example in a regression task, a neural net could be interpreted as a probability function (Gaussian with the output as mean and some constant variance), it is "natural" to form a distance on the manifold under the KL-divergence (and a Fisher information matrix as its metric). </p>
  
  <p>Now, if I want to be creative, I could use the same argument to use "square distance" between the outputs of the neural nets (distance of the means) which I think is not the same as the KL. </p>
  
  <p>Am I wrong, or it is just another legit way? Perhaps, not as good?</p>
  
Tags: <machine-learning><deep-learning>
Title: Why they use KL divergence in Natural gradient?
UserDisplayName: Phizaz
ThreadText: # Why they use KL divergence in Natural gradient?
  
  (Asked by: username_0 on 2019-10-30)
  
  Natural gradient aims to do a steepest descent on the "function" space, a manifold that is independent from how the function is parameterized. It argues that the steepest descent on this function space is not the same as steepest descent on the parameter space. We should favor the former.
  
  Since, for example in a regression task, a neural net could be interpreted as a probability function (Gaussian with the output as mean and some constant variance), it is "natural" to form a distance on the manifold under the KL-divergence (and a Fisher information matrix as its metric).
  
  Now, if I want to be creative, I could use the same argument to use "square distance" between the outputs of the neural nets (distance of the means) which I think is not the same as the KL.
  
  Am I wrong, or it is just another legit way? Perhaps, not as good?
  
  **Tags:** machine-learning, deep-learning
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-10-30. Score: 2
  
  The KL divergence has slightly different interpretations depending on the context. The [related Wikipedia article](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Interpretations) contains a section dedicated to these interpretations. Independently of the interpretation, the KL divergence is always defined as a *specific* function of the [cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy) (which you should be familiar with before attempting to understand the KL divergence) between two distributions (in this case, probability mass functions)
  
  \begin{align}
  D\_\text{KL}(P\parallel Q)
  &= -\sum\_{x\in\mathcal{X}} p(x) \log q(x) + \sum\_{x\in\mathcal{X}} p(x) \log p(x) \\
  &= H(P, Q) - H(P)
  \end{align}
  where $H(P, Q)$ is the cross-entropy of the distribution $P$ and $Q$ and $H(P) = H(P, P)$.
  
  The KL is not a metric, given that it does not obey the triangle inequality. In other words, in general, $D\_\text{KL}(P\parallel Q) \neq D\_\text{KL}(Q\parallel P) $.
  
  *Given that a neural network is trained to output the mean (which can be a scalar or a vector) and the variance (which can be a scalar, a vector or a matrix), why don't we use a metric like the MSE to compare means and variances?* When you use the KL divergence, you don't want to compare just numbers (or matrices), but probability distributions (more precisely, probability densities or mass functions), so you will not compare just the mean and the variance of two different distributions, but you will actually compare the distributions. See [the example of the application of the KL divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Basic_example) in the related Wikipedia article.
  
  > **Comments:**
  > **username_0**: MSE doesn't compare variance, I know, but I could argue that my neural net is not a probabilistic function (it is a deterministic one), hence there is no variance. A natural gradient is then should be formed in MSE distance of the function output?  
  
  ---
  
  ### Answer by username_2 on 2019-10-30. Score: 1
  
  Yes, Squared distances & KL Divergence are not the same. Squared distance between means is not a useful metric as it doesn't gauge the amount of similarity between 2 distributions.
  
  When we compute \begin{align}
  D\_\text{KL}(P\parallel Q)
  \end{align}
  We are computing the amount of information that is lost when we approximate **P** as **Q**. Ideally, we would want the KL divergence to be as low as possible.
  Here is an interesting article <https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained>
  where the author has explained KL Divergence with a toy example.
  
  I hope it helps :)
  
  > **Comments:**
  > **username_0**: It is true if the assumption is that the neural network is a probability function. However, the assumption is arbitrary. I could do equally well if I assume that the neural net is a deterministic function. The square distance makes much more sense now.  
  
  ---
  
  ### Answer by username_3 on 2020-06-26. Score: 0
  
  I have been reading a lot about Natural Gradient and its use to find a descent direction. I found that [this post](https://wiseodd.github.io/techblog/2018/03/14/natural-gradient/) was the most clear.
  
  Consider a model $p$ parameterized by some parameters $\theta$ and we want to maximize the likelihood of observing our data $x$ under this model: $p(x|\theta)$. To optimise this likelihood we can take steps in the the **distribution space**. Updating the parameters $\theta$ we need to measure how our likelihood changes and this is measure using the KLK divergence.
  
  Even though the KL divergence is not a "proper" distance metric as it is not symmetric, it is still quite informative about the similarity between distributions. It's practical because it can capture differences between distributions that the Euclidean metric (parameter-dependent) could not (see [the same post](https://wiseodd.github.io/techblog/2018/03/14/natural-gradient/) for a simple example).
  
  So answering your question is essentially answering which is the best between Natural Gradient Descent and "Normal" Gradient Descent in the Euclidean space where your loss is measured with a L2 norm.
  You can train the same model using both methods and you will just find different descent directions.
  
  Hopefully though, both will converge but in my opinion Natural Gradient descent should be superior in nature. It is just very expensive to actually compute because to find the direction in distribution space you need to compute the inverse Fisher matrix $F^{-1}$ or approximate it and that's quite costly as it is of size $n\times n$ where $n$ is the size of $\theta$ which is typically high in neural networks.
  
  
  ---
  
  