Answers.Body: <p>It’s not an exhaustive answer to your question, but here some aspects that might be helpful:</p>
  
  <p>A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class <span class="math-container">$c_i$</span> where <span class="math-container">$i$</span> is from some index set <span class="math-container">$I$</span>. The class membership is represented by one-hot encoding, which corresponds to a distribution <span class="math-container">$P$</span> with <span class="math-container">$p_i = 1$</span> for the class the data point belongs to and <span class="math-container">$p_j = 0$</span> for all <span class="math-container">$i \neq j$</span>. The loss function mostly used in this problems is the average categorical cross-entropy <span class="math-container">$\langle E_P[-log(Q)]\rangle_{\rm data} = \langle H(P) - D(P || Q)\rangle_{\rm data}$</span>, which in the case of single-class membership reduces to the binary cross-entropy. Because only one probability in <span class="math-container">$P$</span> is non-zero in this case, the cross-entropy simplifies to <span class="math-container">$\langle -\log(q_i)\rangle_{\rm data}$</span>. The reverse KL-Divergence <span class="math-container">$D(Q || P)$</span> is strictly speaking not even defined in this case, as <span class="math-container">$Q$</span> needs to be absolute continuous w.r.t. <span class="math-container">$P$</span>. </p>
  
  <p>There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply in full. From a more conceptual point of view, <a href="https://datascience.stackexchange.com/questions/9302/the-cross-entropy-error-function-in-neural-networks">this thread</a> summaries nicely the relation of cross-entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a typical sample drawn from the model under the (empirical) distribution of the observed data. </p>
  
  <p>A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for <em>each</em> point in the input space, this is why the average cross-entropy is minimized.</p>
  
  <p>Classification problems are just one example of supervised learning, so this answer might not fully cover your question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-06T11:34:47.823Z
Answers.LastActivityDate: 2019-04-08T08:19:02.583Z
Answers.LastEditDate: 2019-04-08T08:19:02.583Z
Answers.UserDisplayName: user1587520
Body: <p>Forward KL Divergence (also known as cross entropy loss) is a standard loss function in supervised learning problems. I understand why it is so: matching a known a trained distribution to a known distribution fits <span class="math-container">$P \log(P/Q)$</span> where <span class="math-container">$P$</span> is the known distribution. </p>
  
  <p>Why isn't the reverse KL divergence commonly used in supervised learning?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-05T19:54:28.293Z
Comments.Text: Maybe say a little more about what reverse KL divergence is.
Comments.UserDisplayName: Philip Raeisghasem
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-08T00:24:28.223Z
Comments.Text: Reverse KL Divergence would be $Q \log (Q/P)$
Comments.UserDisplayName: cgo
Tags: <machine-learning><optimization><objective-functions><supervised-learning>
Title: Why isn't the reverse KL divergence commonly used in supervised learning?
UserDisplayName: cgo
ThreadText: # Why isn't the reverse KL divergence commonly used in supervised learning?
  
  (Asked by: username_0 on 2019-04-05)
  
  Forward KL Divergence (also known as cross entropy loss) is a standard loss function in supervised learning problems. I understand why it is so: matching a known a trained distribution to a known distribution fits $P \log(P/Q)$ where $P$ is the known distribution.
  
  Why isn't the reverse KL divergence commonly used in supervised learning?
  
  **Tags:** machine-learning, optimization, objective-functions, supervised-learning
  
  > **Comments:**
  > **username_1**: Maybe say a little more about what reverse KL divergence is.  
  > **username_0**: Reverse KL Divergence would be $Q \log (Q/P)$  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-04-06. Score: 1
  
  It’s not an exhaustive answer to your question, but here some aspects that might be helpful:
  
  A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class $c\_i$ where $i$ is from some index set $I$. The class membership is represented by one-hot encoding, which corresponds to a distribution $P$ with $p\_i = 1$ for the class the data point belongs to and $p\_j = 0$ for all $i \neq j$. The loss function mostly used in this problems is the average categorical cross-entropy $\langle E\_P[-log(Q)]\rangle\_{\rm data} = \langle H(P) - D(P || Q)\rangle\_{\rm data}$, which in the case of single-class membership reduces to the binary cross-entropy. Because only one probability in $P$ is non-zero in this case, the cross-entropy simplifies to $\langle -\log(q\_i)\rangle\_{\rm data}$. The reverse KL-Divergence $D(Q || P)$ is strictly speaking not even defined in this case, as $Q$ needs to be absolute continuous w.r.t. $P$.
  
  There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply in full. From a more conceptual point of view, [this thread](https://datascience.stackexchange.com/questions/9302/the-cross-entropy-error-function-in-neural-networks) summaries nicely the relation of cross-entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a typical sample drawn from the model under the (empirical) distribution of the observed data.
  
  A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for *each* point in the input space, this is why the average cross-entropy is minimized.
  
  Classification problems are just one example of supervised learning, so this answer might not fully cover your question.
  
  
  ---
  
  