Answers.Body: <p>The quick answer is yes. The implementation of sklearn use both a bias for each weight and a final threshold, by default set to 0.5</p>
  <p>It sounds to me though that you don't have a clear idea about the purpose of the threshold in the first place, so let's elaborate more on that.</p>
  <h3>why do we use a threshold</h3>
  <p>A perceptron, and any other type of model, return continuous values. For some tasks (e.g. regression) we can compute error directly on these continuous values, but for other tasks (e.g. classification) we need to convert these values into discrete values, usually binary, which tell us if the model is predicting a yes or a no for a specific node. The threshold serves precisely for this purpose. We apply it to the very last probabilities returned by a model to convert them into discrete outputs (and I stress out probabilities, not logits, so you need to apply first either sigmoid of softmax).</p>
  <h3>when do we modify the threshold</h3>
  <p>The only moment in which is reasonable to change the threshold is after training. And the reason to do so is that we want to test if there are values different from 0.5 that lead to better metrics score. Usually this is performed using the <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic#:%7E:text=A%20receiver%20operating%20characteristic%20curve,which%20led%20to%20its%20name." rel="nofollow noreferrer">ROC</a> curve. i.e. we compute the amount of false positives an false negatives for different thresholds and we check what threshold lead to the highest <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve" rel="nofollow noreferrer">AUC</a> score (the red one in the plot, which maximize the ROC area).</p>
  <p>But there is no point in using a different threshold from 0.5 during training. And there is no point in associating multiple threshold to each bias in a model. The only reason why the threshold is used is to discretize continuous values ad turn them into final predictions.</p>
  <p><a href="https://i.sstatic.net/o1Vw0.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/o1Vw0.jpg" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/PCnJ9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PCnJ9.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-12T13:54:30.550Z
Answers.Text: Thank you for your answer, but I was wondering why you wrote about threshold during training, as I thought the cost function directly used the output of the final activation function, without threshold.
Answers.UserDisplayName: Leosa99 _
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-12T12:44:20.567Z
Answers.LastActivityDate: 2022-03-12T12:44:20.567Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>In the <a href="https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/linear_model/_base.py#L386-L430" rel="nofollow noreferrer">sklearn code of the Perceptron</a> you can see <a href="https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/linear_model/_base.py#L427" rel="nofollow noreferrer">here</a> that a threshold of 0 is used when calling the <code>predict</code> function. And <a href="https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/linear_model/_base.py#L408" rel="nofollow noreferrer">here</a> is where the bias is added (they call the bias <code>_intercept</code>).</p>
  <p>Using both doesn't really change the behavior of the perceptron, because you can reformulate the formulation like this:</p>
  <p><span class="math-container">$$
  \begin{align}
  \mathbf{Wx} + b &amp;&gt; t &amp;&amp; | \text{ Perceptron with bias $b$ and threshold $t$}\\
  \Leftrightarrow \mathbf{Wx}+(b-t) &amp;&gt; 0 &amp;&amp; | \text{ threshold 'included' in bias} \\
  \Leftrightarrow \mathbf{Wx}       &amp;&gt; (t - b) &amp;&amp; | \text{ bias 'included' in threshold}
  \end{align}
  $$</span></p>
  <p>(From a learning perspective, there is a slight difference, because the threshold is not a learnable parameter, so it would change the optimum of the weights.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-12T12:12:45.740Z
Answers.LastActivityDate: 2022-03-12T12:12:45.740Z
Answers.UserDisplayName: Chillston
Body: <p>Reading this <a href="https://ai.stackexchange.com/questions/11684/can-a-neuron-have-both-a-bias-and-a-threshold">Can a neuron have both a bias and a threshold?</a> has confused me, as it appears to be more common to use a threshold of 0 when using bias. But reading this <a href="https://stackoverflow.com/questions/19984957/scikit-learn-predict-default-threshold">https://stackoverflow.com/questions/19984957/scikit-learn-predict-default-threshold</a> indicates that the threshold is 0.5.</p>
  <p>So my question is, is sklearn using both a threshold and a bias term ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-12T13:00:20.100Z
Comments.Text: Questions about software libraries or APIs are off-topic here. You're asking about sklearn, so your question is off-topic.
Comments.UserDisplayName: nbro
Tags: <scikit-learn><single-layer-perceptron>
Title: Is sklearn using both a threshold and a bias term?
UserDisplayName: Leosa99 _
ThreadText: # Is sklearn using both a threshold and a bias term?
  
  (Asked by: username_0 on 2022-03-12)
  
  Reading this [Can a neuron have both a bias and a threshold?](https://ai.stackexchange.com/questions/11684/can-a-neuron-have-both-a-bias-and-a-threshold) has confused me, as it appears to be more common to use a threshold of 0 when using bias. But reading this <https://stackoverflow.com/questions/19984957/scikit-learn-predict-default-threshold> indicates that the threshold is 0.5.
  
  So my question is, is sklearn using both a threshold and a bias term ?
  
  **Tags:** scikit-learn, single-layer-perceptron
  
  > **Comments:**
  > **username_1**: Questions about software libraries or APIs are off-topic here. You're asking about sklearn, so your question is off-topic.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-03-12. Score: 0 | Accepted answer
  
  The quick answer is yes. The implementation of sklearn use both a bias for each weight and a final threshold, by default set to 0.5
  
  It sounds to me though that you don't have a clear idea about the purpose of the threshold in the first place, so let's elaborate more on that.
  
  ### why do we use a threshold
  
  A perceptron, and any other type of model, return continuous values. For some tasks (e.g. regression) we can compute error directly on these continuous values, but for other tasks (e.g. classification) we need to convert these values into discrete values, usually binary, which tell us if the model is predicting a yes or a no for a specific node. The threshold serves precisely for this purpose. We apply it to the very last probabilities returned by a model to convert them into discrete outputs (and I stress out probabilities, not logits, so you need to apply first either sigmoid of softmax).
  
  ### when do we modify the threshold
  
  The only moment in which is reasonable to change the threshold is after training. And the reason to do so is that we want to test if there are values different from 0.5 that lead to better metrics score. Usually this is performed using the [ROC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#:%7E:text=A%20receiver%20operating%20characteristic%20curve,which%20led%20to%20its%20name.) curve. i.e. we compute the amount of false positives an false negatives for different thresholds and we check what threshold lead to the highest [AUC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve) score (the red one in the plot, which maximize the ROC area).
  
  But there is no point in using a different threshold from 0.5 during training. And there is no point in associating multiple threshold to each bias in a model. The only reason why the threshold is used is to discretize continuous values ad turn them into final predictions.
  
  [![enter image description here](https://i.sstatic.net/o1Vw0.jpg)](https://i.sstatic.net/o1Vw0.jpg)
  [![enter image description here](https://i.sstatic.net/PCnJ9.png)](https://i.sstatic.net/PCnJ9.png)
  
  > **Comments:**
  > **username_0**: Thank you for your answer, but I was wondering why you wrote about threshold during training, as I thought the cost function directly used the output of the final activation function, without threshold.  
  
  ---
  
  ### Answer by username_3 on 2022-03-12. Score: 0
  
  In the [sklearn code of the Perceptron](https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/linear_model/_base.py#L386-L430) you can see [here](https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/linear_model/_base.py#L427) that a threshold of 0 is used when calling the `predict` function. And [here](https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/linear_model/_base.py#L408) is where the bias is added (they call the bias `_intercept`).
  
  Using both doesn't really change the behavior of the perceptron, because you can reformulate the formulation like this:
  
  $$
  \begin{align}
  \mathbf{Wx} + b &> t && | \text{ Perceptron with bias $b$ and threshold $t$}\\
  \Leftrightarrow \mathbf{Wx}+(b-t) &> 0 && | \text{ threshold 'included' in bias} \\
  \Leftrightarrow \mathbf{Wx} &> (t - b) && | \text{ bias 'included' in threshold}
  \end{align}
  $$
  
  (From a learning perspective, there is a slight difference, because the threshold is not a learnable parameter, so it would change the optimum of the weights.)
  
  
  ---
  
  