Answers.Body: <p>@DuttaA has pretty much mentioned the two most appropriate approaches to having this facility. Either the penalty of false positives should be high or the learning rate for the correct class should be high. </p>
  
  <p>I'll give two real-life examples to help you understand it better. </p>
  
  <p>Say you have to teach a teen that substance abuse is injurious to health (eg. Frequent smoking is a <strong>negative</strong> habit). But the teen ends up learning from high effects of the drugs that it is good (<strong>false positive</strong>) and gets addicted to it. You would strictly want to avoid this kind of a situation (<strong>false positive error having a very big cost as compared to false negative error</strong>).</p>
  
  <p>In general, to model the situation when the costs are different, we picture a cost matrix. For a two-class classification problem, the cost matrix would look like:</p>
  
  <p><a href="https://i.sstatic.net/HaVxo.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HaVxo.png" alt="Cost Matrix"></a>
  (courtesy: <a href="http://albahnsen.com/CostSensitiveClassification" rel="nofollow noreferrer">http://albahnsen.com/CostSensitiveClassification</a>)</p>
  
  <p>Now, when designing your cost function, you would want to take into account the weight corresponding to each of the situation. A simple python code would be as follows:</p>
  
  <pre><code>def weighted_cost(pred, act):
      if pred==P and act==P:
          return C_TP * cost(pred, act)
      if pred==P and act==N:
          return C_FP * cost(pred, act)
      if pred==N and act==P:
          return C_FN * cost(pred, act)
      if pred==N and act==N:
          return C_TN * cost(pred, act)
  </code></pre>
  
  <p>Where, <strong>pred</strong> is the predicted class and <strong>act</strong> is the actual class. Here, <strong>C_TP, C_FP, C_TN, C_FN</strong> represent the weights of true positive, false positive etc. The <strong>cost(pred, act)</strong> function will calculate the loss of one training example. You would want to use the weighted_cost function for finding the loss after one training example.</p>
  
  <p>The second approach that @DuttaA mentioned was to vary the learning rate. In real life, you can relate this to the situation when you were asked to write a word <strong>multiple times</strong> if you forget its spelling so that you remember it better. In a way, you learn the correct spelling of the word.</p>
  
  <p>Here, increasing the value of the learning rate <em>(say 4 x alpha)</em> for a class can be viewed as updating the value of the weights <strong>multiple times</strong> <em>(4 times)</em> with the old learning rate <em>(alpha)</em>, similar to what we do by writing the correct spelling of the word multiple times. So, a more important class (in your case it will be the <strong>Negative Class</strong>) should be given more alpha, because a false positive (misclassification of the negative class) has a high penalty. You learn to recognize the correct (negative) class by learning it more number of times (as in the case of learning the spelling of the word).</p>
  
  <p>Let me know if you need any further clarification.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-24T17:02:24.067Z
Answers.LastActivityDate: 2018-04-29T09:29:35.260Z
Answers.LastEditDate: 2018-04-29T09:29:35.260Z
Answers.UserDisplayName: Naveen Venkat
Answers.Body: <p>A funky way of doing this with less overhead is to just over fit the data up to some degree.
  The reason is when you try to over fit the data with the classifier the classification bound tends to wrap around the clusters very tightly and with that model you can some times miss classify positive classes as negative(due to high variance) but there are comparatively less situations where you end up miss classifying negative classes as positive.
  The level of overfitting that needs to be performed is just based on your FP and FN trade off.</p>
  
  <p>I don't think this as a permanent fix but can come handy up to some extent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-08T16:55:01.737Z
Answers.LastActivityDate: 2018-06-08T16:55:01.737Z
Answers.UserDisplayName: koushik
Answers.Body: <p>One more idea - I recall learning about Neyman-Pearson task in my studies. It is a statistical learning method for binary classification problem where overlooked danger (false negative error) is much unwanted. </p>
  
  <p>You set a desired threshold for false negative error rate and then minimize false positive error. You just need to measure conditional probabilities of each class. It may be expressed as a linear program and solved to get the optimal strategy for the threshold of your choice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-22T13:19:49.803Z
Answers.LastActivityDate: 2019-12-22T13:19:49.803Z
Answers.UserDisplayName: V.Mach
Answers.Body: <p>You could vary an error coefficient in training. For example, if the expected output was negative and it gave some value in the positive you can train on C * ERROR and conversely if the expected output was positive and it gave some value in the negative you can train on just the error, that way false positives have more impact on the model as opposed to false negatives.</p>
  <p>Varying learning rates could as well help, however, increasing the learning rate and increase the error have different effects because changing the error will change the direction of the gradient whereas changing the learning rate will only change the gradient's magnitude of effect on the network, two slightly different things.</p>
  <p>(for the learning rate, split the data into two, positive and negative, then train them separately with the learning rate for negative cases larger than for positive cases)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-26T15:49:13.163Z
Answers.LastActivityDate: 2020-06-26T15:49:13.163Z
Answers.UserDisplayName: Andar Manik
Body: <p>I have a binary classification problem, where a false positive error has a very big cost compared to the false negative error. </p>
  
  <p>Is there a way to design a classifier for such problems (preferably, with an implementation of the algorithm)?  </p>
  
Tags: <neural-networks><classification>
Title: Binary classifier that minimizes false positive error
UserDisplayName: ayyoob imani
ThreadText: # Binary classifier that minimizes false positive error
  
  (Asked by: username_0 on 2017-12-15)
  
  I have a binary classification problem, where a false positive error has a very big cost compared to the false negative error.
  
  Is there a way to design a classifier for such problems (preferably, with an implementation of the algorithm)?
  
  **Tags:** neural-networks, classification
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2017-12-24. Score: 3
  
  @DuttaA has pretty much mentioned the two most appropriate approaches to having this facility. Either the penalty of false positives should be high or the learning rate for the correct class should be high. 
  I'll give two real-life examples to help you understand it better. 
  Say you have to teach a teen that substance abuse is injurious to health (eg. Frequent smoking is a negative habit). But the teen ends up learning from high effects of the drugs that it is good (false positive) and gets addicted to it. You would strictly want to avoid this kind of a situation (false positive error having a very big cost as compared to false negative error).
  In general, to model the situation when the costs are different, we picture a cost matrix. For a two-class classification problem, the cost matrix would look like:
  
  (courtesy: http://albahnsen.com/CostSensitiveClassification)
  Now, when designing your cost function, you would want to take into account the weight corresponding to each of the situation. A simple python code would be as follows:
  def weighted_cost(pred, act):
      if pred==P and act==P:
          return C_TP * cost(pred, act)
      if pred==P and act==N:
          return C_FP * cost(pred, act)
      if pred==N and act==P:
          return C_FN * cost(pred, act)
      if pred==N and act==N:
          return C_TN * cost(pred, act)
  
  Where, pred is the predicted class and act is the actual class. Here, C_TP, C_FP, C_TN, C_FN represent the weights of true positive, false positive etc. The cost(pred, act) function will calculate the loss of one training example. You would want to use the weighted_cost function for finding the loss after one training example.
  The second approach that @DuttaA mentioned was to vary the learning rate. In real life, you can relate this to the situation when you were asked to write a word multiple times if you forget its spelling so that you remember it better. In a way, you learn the correct spelling of the word.
  Here, increasing the value of the learning rate (say 4 x alpha) for a class can be viewed as updating the value of the weights multiple times (4 times) with the old learning rate (alpha), similar to what we do by writing the correct spelling of the word multiple times. So, a more important class (in your case it will be the Negative Class) should be given more alpha, because a false positive (misclassification of the negative class) has a high penalty. You learn to recognize the correct (negative) class by learning it more number of times (as in the case of learning the spelling of the word).
  Let me know if you need any further clarification.
  
  
  ---
  
  ### Answer by username_2 on 2018-06-08. Score: 1
  
  A funky way of doing this with less overhead is to just over fit the data up to some degree.
  The reason is when you try to over fit the data with the classifier the classification bound tends to wrap around the clusters very tightly and with that model you can some times miss classify positive classes as negative(due to high variance) but there are comparatively less situations where you end up miss classifying negative classes as positive.
  The level of overfitting that needs to be performed is just based on your FP and FN trade off.
  
  I don't think this as a permanent fix but can come handy up to some extent.
  
  
  ---
  
  ### Answer by username_3 on 2019-12-22. Score: 0
  
  One more idea - I recall learning about Neyman-Pearson task in my studies. It is a statistical learning method for binary classification problem where overlooked danger (false negative error) is much unwanted.
  
  You set a desired threshold for false negative error rate and then minimize false positive error. You just need to measure conditional probabilities of each class. It may be expressed as a linear program and solved to get the optimal strategy for the threshold of your choice.
  
  
  ---
  
  ### Answer by username_4 on 2020-06-26. Score: 0
  
  You could vary an error coefficient in training. For example, if the expected output was negative and it gave some value in the positive you can train on C \* ERROR and conversely if the expected output was positive and it gave some value in the negative you can train on just the error, that way false positives have more impact on the model as opposed to false negatives.
  
  Varying learning rates could as well help, however, increasing the learning rate and increase the error have different effects because changing the error will change the direction of the gradient whereas changing the learning rate will only change the gradient's magnitude of effect on the network, two slightly different things.
  
  (for the learning rate, split the data into two, positive and negative, then train them separately with the learning rate for negative cases larger than for positive cases)
  
  
  ---
  
  