Answers.Body: <p>The vector functions for true positive, false positive etc all make use of the "magic" numbers <span class="math-container">$0$</span> and <span class="math-container">$1$</span> used to represent Boolean values. They are convenience methods that you can use in a numerical library, but you do need to be aware of the fundamental Boolean nature of the data. The <span class="math-container">$0$</span> and <span class="math-container">$1$</span> values allow the maths for calculating TP et al, but are not fundamental to it, they are a representation.</p>
  
  <p>Your derivations of gradients for the functions you give seem correct, barring the odd typo. However, the gradient doesn't really apply to the value of <span class="math-container">$\mathbf{y}$</span>, because all components of <span class="math-container">$\mathbf{y}$</span> are either <span class="math-container">$0$</span> or <span class="math-container">$1$</span>. The idea that you could increase <span class="math-container">$\mathbf{y}$</span> slightly where  <span class="math-container">$\mathbf{\hat{y}}$</span> is <span class="math-container">$1$</span> in order to increase the value of the TP metric slightly has no basis. Instead the only valid changes to make an improvement are to modify FN values from <span class="math-container">$0$</span> to <span class="math-container">$1$</span> exactly.</p>
  
  <p>You could probably still use your derivations as a gradient for optimisation (it would not be the only time in machine learning that something does not quite apply theoretically but you could still use it in practice). However, you then immediately hit the problem of how the values of <span class="math-container">$\mathbf{y}$</span> have been discretised to  <span class="math-container">$0$</span> or <span class="math-container">$1$</span> - <em>that</em> function will not be differentiable, and it will prevent you back propagating your gradients to the neural network weights that you want to change. If you fix that follow-on problem using a smoother function (e.g. a sigmoid) then you are likely to end up with something close to either cross-entropy loss or the perceptron update step. </p>
  
  <p>In other words, although what you have been told is an over-simplification, you will not find a way to improve the performance of your classifier by adding cost functions based directly on TP, FP etc. That is what binary cross-entropy loss is already doing. There are other, perhaps more fruitful avenues of investigation - hyperparameter searches, regularisation, ensembling, and if you have an unbalanced data set then consider weighting the true/false costs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-30T08:40:19.227Z
Answers.Text: If I use the precision as cost function (TP/(TP+FN)) and add a sigmoid function to the inputs, does it have chance to work (not outperform the classical function like cross-entropy but simply have a better result than a random system) ?
Answers.UserDisplayName: Léonard Barras
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-30T08:43:15.787Z
Answers.Text: @LéonardBarras: That will depend on other details of your design, which I don't know and would be too complex to analyse in comments or extend the current question. Intuitively, I think you will at best end up re-inventing cross-entropy, but you might also end up with something that sort of works but that is strictly worse than cross-entropy. If you want to know for sure, then build it and test it.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-29T07:56:38.660Z
Answers.LastActivityDate: 2019-08-29T08:12:00.430Z
Answers.LastEditDate: 2019-08-29T08:12:00.430Z
Answers.UserDisplayName: Neil Slater
Body: <p>I wanted to use True Positive (and True Negative) in my cost function to make to modify the ROC shape of my classifier. Someone told me and I read that it is not differentiable and therefore not usable as a cost function for a neural network.</p>
  
  <p>In the example where 1 is positive and 0 negative I deduce the following equation for True Positive (<span class="math-container">$\hat y = prediction, y = label$</span>):</p>
  
  <p><span class="math-container">$$ TP = \bf(\hat{y}^Ty) $$</span>
  <span class="math-container">$$ \frac{\partial TP}{\partial \bf y} = \bf y $$</span></p>
  
  <p>The following for True Negative:
  <span class="math-container">$$ TN = \bf(\hat{y}-1)^T(y-1) $$</span>
  <span class="math-container">$$ \frac{\partial TN}{\partial \bf y} = \bf \hat y^T -1    $$</span></p>
  
  <p>The False Positive: 
  <span class="math-container">$$ FP = - \bf  (\hat y^T-1) y $$</span>
  <span class="math-container">$$ \frac{\partial FP}{\partial \bf y} = - \bf ( \hat y^T - 1)    $$</span></p>
  
  <p>The False Negative: 
  <span class="math-container">$$ FN = \bf  \hat y^T (y-1) $$</span>
  <span class="math-container">$$ \frac{\partial FN}{\partial \bf y} = \bf \hat y    $$</span></p>
  
  <p>All equations seem differentiable to me. Can someone explain where I went wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-29T07:31:03.990Z
Comments.Text: As an aside, it is more usual to have $\hat{y}$ as the prediction and $y$ as the true value.
Comments.UserDisplayName: Neil Slater
Tags: <deep-learning><metric>
Title: Using True Positive as a Cost Function
UserDisplayName: Léonard Barras
ThreadText: # Using True Positive as a Cost Function
  
  (Asked by: username_0 on 2019-08-28)
  
  I wanted to use True Positive (and True Negative) in my cost function to make to modify the ROC shape of my classifier. Someone told me and I read that it is not differentiable and therefore not usable as a cost function for a neural network.
  
  In the example where 1 is positive and 0 negative I deduce the following equation for True Positive ($\hat y = prediction, y = label$):
  
  $$ TP = \bf(\hat{y}^Ty) $$
  $$ \frac{\partial TP}{\partial \bf y} = \bf y $$
  
  The following for True Negative:
  $$ TN = \bf(\hat{y}-1)^T(y-1) $$
  $$ \frac{\partial TN}{\partial \bf y} = \bf \hat y^T -1 $$
  
  The False Positive:
  $$ FP = - \bf (\hat y^T-1) y $$
  $$ \frac{\partial FP}{\partial \bf y} = - \bf ( \hat y^T - 1) $$
  
  The False Negative:
  $$ FN = \bf \hat y^T (y-1) $$
  $$ \frac{\partial FN}{\partial \bf y} = \bf \hat y $$
  
  All equations seem differentiable to me. Can someone explain where I went wrong?
  
  **Tags:** deep-learning, metric
  > **Comments:**
  > **username_1**: As an aside, it is more usual to have $\hat{y}$ as the prediction and $y$ as the true value.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-29. Score: 1
  
  The vector functions for true positive, false positive etc all make use of the "magic" numbers $0$ and $1$ used to represent Boolean values. They are convenience methods that you can use in a numerical library, but you do need to be aware of the fundamental Boolean nature of the data. The $0$ and $1$ values allow the maths for calculating TP et al, but are not fundamental to it, they are a representation.
  
  Your derivations of gradients for the functions you give seem correct, barring the odd typo. However, the gradient doesn't really apply to the value of $\mathbf{y}$, because all components of $\mathbf{y}$ are either $0$ or $1$. The idea that you could increase $\mathbf{y}$ slightly where $\mathbf{\hat{y}}$ is $1$ in order to increase the value of the TP metric slightly has no basis. Instead the only valid changes to make an improvement are to modify FN values from $0$ to $1$ exactly.
  
  You could probably still use your derivations as a gradient for optimisation (it would not be the only time in machine learning that something does not quite apply theoretically but you could still use it in practice). However, you then immediately hit the problem of how the values of $\mathbf{y}$ have been discretised to $0$ or $1$ - *that* function will not be differentiable, and it will prevent you back propagating your gradients to the neural network weights that you want to change. If you fix that follow-on problem using a smoother function (e.g. a sigmoid) then you are likely to end up with something close to either cross-entropy loss or the perceptron update step.
  
  In other words, although what you have been told is an over-simplification, you will not find a way to improve the performance of your classifier by adding cost functions based directly on TP, FP etc. That is what binary cross-entropy loss is already doing. There are other, perhaps more fruitful avenues of investigation - hyperparameter searches, regularisation, ensembling, and if you have an unbalanced data set then consider weighting the true/false costs.
  
  > **Comments:**
  > **username_0**: If I use the precision as cost function (TP/(TP+FN)) and add a sigmoid function to the inputs, does it have chance to work (not outperform the classical function like cross-entropy but simply have a better result than a random system) ?  
  > **username_1**: @LéonardBarras: That will depend on other details of your design, which I don't know and would be too complex to analyse in comments or extend the current question. Intuitively, I think you will at best end up re-inventing cross-entropy, but you might also end up with something that sort of works but that is strictly worse than cross-entropy. If you want to know for sure, then build it and test it.  
  
  ---
  
  