Answers.Body: <p>For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate <span class="math-container">$\hat{p}$</span> is within some ratio of the true value. So an estimate of <span class="math-container">$\hat{p} = 0.1$</span> is a better estimate if the true value is <span class="math-container">$p = 0.2$</span> than if the true value is <span class="math-container">$p = 0.01$</span> (even though the latter value is closer and would score better under MSE). It also applies that you care that 0.9 is by the same logic "closer" to 0.8 than it is to 0.97. With cross-entropy loss, extreme confidence (predicting close to <span class="math-container">$0$</span> or close to <span class="math-container">$1$</span>) is penalised more heavily when it is wrong.</p>
  
  <p>For completeness, the loss function (per data point) is:</p>
  
  <p><span class="math-container">$$\mathcal{L}(\hat{y},y) = -(y\text{log}(\hat{y})+ (1-y)\text{log}(1-\hat{y}))$$</span></p>
  
  <p>This is likely to be the same loss function as you are using for your objective (or at least it should be), so for tests, simply also use it as your metric*.</p>
  
  <p><span class="math-container">$\hat{y}$</span> is your predicted probability of being in class A, and can be in range <span class="math-container">$[0,1]$</span>. Ideally you have ground truth probabilities and <span class="math-container">$y$</span> is also in that range. In which case the only problem is that the "perfect" score is no longer <span class="math-container">$0$</span> but some positive number. If that bothers you, then you could offset by the perfect score, pre-calculating it on each data set (just set <span class="math-container">$\hat{y} = y$</span> for each item and you will find the minimum possible score).</p>
  
  <p>If you don't have ground truth probabilities, but you do have classes, then <span class="math-container">$y$</span> will either be 0 or 1, and the metric still works. To get an accurate metric, you will need enough samples that the relative frequencies of each class depending on input has a significant effect. That is, you need more data, both training and test, in order to train for accurate probabilities instead of targeting simpler classification accuracy metrics.</p>
  
  <p>Similar logic also works for multi-class probabilities. However, many off-the-shelf libraries use an optimisation in the loss function - assuming only one true class - which makes using probabilities as ground truth impossible. You might therefore need to write your own loss function and gradient functions based on multi-class cross-entropy loss in that case.</p>
  
  <hr>
  
  <p>* I am making the assumption here that you use standard conventions for noting loss functions (typically per item), cost functions (typically aggregated across a data set and possibly multiple loss functions) and metric functions which don't have to be differentiable or usable as either of the former. The cost function is usually also fed into optimisers as the objective function - i.e. it has the important job of driving parameters to reach a maximum or minimum value. For gradient based solvers, that means it must be differentiable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-05T08:11:06.013Z
Answers.LastActivityDate: 2019-07-05T08:48:31.723Z
Answers.LastEditDate: 2019-07-05T08:48:31.723Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm aware of metrics like accuracy (correct predictions / total predictions) for models that classify things. However, I'm working on a model that outputs the probability of a datapoint belonging to one of two classes. What metrics can/should be used to evaluate these types of models?</p>
  
  <p>I'm currently using mean squared error, but I would like to know if there are other metrics, and what the advantages/disadvantages of those metrics are.</p>
  
Tags: <cross-validation><metric>
Title: Metrics for evaluating models that output probabilities
UserDisplayName: Shylock
ThreadText: # Metrics for evaluating models that output probabilities
  
  (Asked by: username_0 on 2019-07-05)
  
  I'm aware of metrics like accuracy (correct predictions / total predictions) for models that classify things. However, I'm working on a model that outputs the probability of a datapoint belonging to one of two classes. What metrics can/should be used to evaluate these types of models?
  
  I'm currently using mean squared error, but I would like to know if there are other metrics, and what the advantages/disadvantages of those metrics are.
  
  **Tags:** cross-validation, metric
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-05. Score: 1
  
  For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within some ratio of the true value. So an estimate of $\hat{p} = 0.1$ is a better estimate if the true value is $p = 0.2$ than if the true value is $p = 0.01$ (even though the latter value is closer and would score better under MSE). It also applies that you care that 0.9 is by the same logic "closer" to 0.8 than it is to 0.97. With cross-entropy loss, extreme confidence (predicting close to $0$ or close to $1$) is penalised more heavily when it is wrong.
  
  For completeness, the loss function (per data point) is:
  
  $$\mathcal{L}(\hat{y},y) = -(y\text{log}(\hat{y})+ (1-y)\text{log}(1-\hat{y}))$$
  
  This is likely to be the same loss function as you are using for your objective (or at least it should be), so for tests, simply also use it as your metric\*.
  
  $\hat{y}$ is your predicted probability of being in class A, and can be in range $[0,1]$. Ideally you have ground truth probabilities and $y$ is also in that range. In which case the only problem is that the "perfect" score is no longer $0$ but some positive number. If that bothers you, then you could offset by the perfect score, pre-calculating it on each data set (just set $\hat{y} = y$ for each item and you will find the minimum possible score).
  
  If you don't have ground truth probabilities, but you do have classes, then $y$ will either be 0 or 1, and the metric still works. To get an accurate metric, you will need enough samples that the relative frequencies of each class depending on input has a significant effect. That is, you need more data, both training and test, in order to train for accurate probabilities instead of targeting simpler classification accuracy metrics.
  
  Similar logic also works for multi-class probabilities. However, many off-the-shelf libraries use an optimisation in the loss function - assuming only one true class - which makes using probabilities as ground truth impossible. You might therefore need to write your own loss function and gradient functions based on multi-class cross-entropy loss in that case.
  
  ---
  
  \* I am making the assumption here that you use standard conventions for noting loss functions (typically per item), cost functions (typically aggregated across a data set and possibly multiple loss functions) and metric functions which don't have to be differentiable or usable as either of the former. The cost function is usually also fed into optimisers as the objective function - i.e. it has the important job of driving parameters to reach a maximum or minimum value. For gradient based solvers, that means it must be differentiable.
  
  
  ---
  
  