Answers.Body: <p>Actually, the <a href="https://en.wikipedia.org/wiki/Cross_entropy" rel="noreferrer">cross-entropy</a> loss function would be appropriate here, since it measures the &quot;distance&quot; between a distribution <span class="math-container">$q$</span> and the &quot;true&quot; distribution <span class="math-container">$p$</span>.</p>
  <p>You are right, though, that using a loss function called &quot;cross_entropy&quot; in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,</p>
  <p><span class="math-container">$$H(p,q)=-\sum_{x\in X} p(x) \log q(x).$$</span>
  <span class="math-container">$ $</span></p>
  <p>Note that one-hot labels would mean that
  <span class="math-container">$$
  p(x) = 
  \begin{cases}
  1 &amp; \text{if }x \text{ is the true label}\\
  0 &amp; \text{otherwise} 
  \end{cases}$$</span></p>
  <p>which causes the cross-entropy <span class="math-container">$H(p,q)$</span> to reduce to the form you're familiar with:</p>
  <p><span class="math-container">$$H(p,q) = -\log q(x_{label})$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-14T22:38:31.020Z
Answers.LastActivityDate: 2019-04-14T22:38:31.020Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Philip Raeisghasem
Body: <p>What loss function is most appropriate when training a model with target values that are probabilities? For example, I have a 3-output model. I want to train it with a feature vector <span class="math-container">$x=[x_1, x_2, \dots, x_N]$</span> and a target <span class="math-container">$y=[0.2, 0.3, 0.5]$</span>. </p>
  
  <p>It seems like something like cross-entropy doesn't make sense here since it assumes that a single target is the correct label.</p>
  
  <p>Would something like MSE (after applying softmax) make sense, or is there a better loss function?</p>
  
Tags: <neural-networks><machine-learning><objective-functions><probability-distribution>
Title: What loss function to use when labels are probabilities?
UserDisplayName: Thomas Johnson
ThreadText: # What loss function to use when labels are probabilities?
  
  (Asked by: username_0 on 2019-04-14)
  
  What loss function is most appropriate when training a model with target values that are probabilities? For example, I have a 3-output model. I want to train it with a feature vector $x=[x\_1, x\_2, \dots, x\_N]$ and a target $y=[0.2, 0.3, 0.5]$.
  
  It seems like something like cross-entropy doesn't make sense here since it assumes that a single target is the correct label.
  
  Would something like MSE (after applying softmax) make sense, or is there a better loss function?
  
  **Tags:** neural-networks, machine-learning, objective-functions, probability-distribution
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-04-14. Score: 10 | Accepted answer
  
  Actually, the [cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy) loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$.
  
  You are right, though, that using a loss function called "cross\_entropy" in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,
  
  $$H(p,q)=-\sum\_{x\in X} p(x) \log q(x).$$
  $ $
  
  Note that one-hot labels would mean that
  $$
  p(x) =
  \begin{cases}
  1 & \text{if }x \text{ is the true label}\\
  0 & \text{otherwise}
  \end{cases}$$
  
  which causes the cross-entropy $H(p,q)$ to reduce to the form you're familiar with:
  
  $$H(p,q) = -\log q(x\_{label})$$
  
  
  ---
  
  