Answers.Body: <p>Since <span class="math-container">$g$</span> is your logistic regression model, over data samples <span class="math-container">$x$</span>, the output of <span class="math-container">$g(x)$</span> is a scalar value between <span class="math-container">$0$</span> and <span class="math-container">$1$</span> that is usually interpreted as a <em>probability value</em>.</p>
  <ul>
  <li>We have that <span class="math-container">$g(x)=\sigma(W x + b)$</span> essentially, where <span class="math-container">$W,b$</span> are the parameters that you learn and <span class="math-container">$\sigma$</span> is the sigmoid squashing function (the one that keeps everything in <span class="math-container">$[0, 1]$</span>).</li>
  <li>Furthermore, <span class="math-container">$g(x)=p(y=1\mid x)$</span> meaning that <span class="math-container">$g(x)$</span> represents the probability that the sample <span class="math-container">$x$</span> belongs to the <em>positive</em> or, as the book says in page 49, <em>most probable class</em> - as a side note, this is something that you can change and usually define before learning.</li>
  </ul>
  <p>Now, let's assume that the two classes <span class="math-container">$y=1$</span> (the positive or most probable) and <span class="math-container">$y=0$</span> (the negative or less probable) have real-world probabilities of: <span class="math-container">$p(y=1\mid x) = 0.7$</span> and <span class="math-container">$p(y=0\mid x) = 1 - p(y=1\mid x) = 0.3$</span>. Let's also be realistic, in the sense that the two class distributions overlap: this implies that you have some misclassification errors due to the fact that is not possible to separate the two classes perfectly. Note: there could be many reasons for this, such as mis-labelling, and similarity of data samples between the two classes.</p>
  <p>At this point, we have <span class="math-container">$g(x)=p(y=1\mid x)$</span> that is the probability of <span class="math-container">$x$</span> to belong to the positive class (<span class="math-container">$y=1$</span>), implying that there is a <span class="math-container">$70\%$</span> chance that <span class="math-container">$x$</span> has label <span class="math-container">$1$</span>. In order to perform a binary classification, you need to get rid of the probabilities by <em>thresholding them</em>, such as: <span class="math-container">$\hat{y}(x) = g(x) &gt; t$</span>.</p>
  <ul>
  <li>If <span class="math-container">$g(x)&gt;t$</span> you assign label <span class="math-container">$1$</span> (so <span class="math-container">$\hat{y}(x)=1$</span>), otherwise you assign label <span class="math-container">$0$</span>.</li>
  <li>In case the model <span class="math-container">$g$</span> predicts the real-world class probabilities correctly (i.e., <span class="math-container">$p(y=1)=0.7$</span> and <span class="math-container">$p(y=0)=0.3$</span>) - as assumed by the book - you have that if the threshold is chosen to be <span class="math-container">$t=0.5$</span> (or <span class="math-container">$r=0.5$</span> as called in the book), happens that <span class="math-container">$g(x)&gt;0.5$</span> minimizes the number of misclassifications, equivalently to maximize the accuracy of predictions.</li>
  <li>In other words, for a threshold of <span class="math-container">$0.5$</span> you get maximum accuracy. But in real classification problems you care also about other measures, such as the AUC of ROC/PR curves, FPR and TPR, F1-score, etc. Basically, you can target a metric and tune the threshold <span class="math-container">$t$</span> to get the rate of error that best suits you application.</li>
  </ul>
  <p>If still you have doubts about your question:</p>
  <blockquote>
  <p>if the model provides a correct description of the real-world class probabilities, why do we have misclassifications?</p>
  </blockquote>
  <p>Consider a classifier <span class="math-container">$g'(x)=0.7$</span> that always outputs a probability of <span class="math-container">$0.7$</span> regardless the data point, <span class="math-container">$x$</span>. Even if the true class probability is <span class="math-container">$0.7$</span>, this model is still erroneous <span class="math-container">$30\%$</span> of the times when classifying all the <span class="math-container">$x'$</span> such that <span class="math-container">$y(x') = 0$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-11T21:28:12.517Z
Answers.LastActivityDate: 2023-11-13T13:52:25.643Z
Answers.LastEditDate: 2023-11-13T13:52:25.643Z
Answers.UserDisplayName: Luca Anzalone
Answers.Body: <p>Note your book specifically claims 'the model provides a correct description of the real-world class <em>probabilities</em>', which exactly means the <a href="https://en.wikipedia.org/wiki/Logistic_regression#Many_explanatory_variables,_two_categories" rel="nofollow noreferrer">multi-explanatory variables logistic function</a> <span class="math-container">$g(\mathbf{x})$</span> trained in your logistic binary classifier <span class="math-container">$\hat{y}(\mathbf{x})$</span>, as a special type of S-shaped sigmoid function transforming the linear combination of input features into a <em>probability</em> between 0 and 1 could be possibly correct in the real world. And if this is true, then intuitively <span class="math-container">$r=0.5$</span> gives the best overall accuracy and the smallest possible number of misclassifications on average.</p>
  <p>In summary it's the logistic model <span class="math-container">$g(\mathbf{x})$</span> could possibly provide a correct description of the real-world class probabilities, but even this is the case, the resulting binary classifier <span class="math-container">$\hat{y}(\mathbf{x})$</span> would still possibly have misclassification since its accuracy may not be 100% either on your training samples or unseen test samples.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-11T22:21:56.397Z
Answers.LastActivityDate: 2023-11-11T23:54:09.997Z
Answers.LastEditDate: 2023-11-11T23:54:09.997Z
Answers.UserDisplayName: cinch
Body: <p>I am reading  the book: MACHINE LEARNING- A First Course for Engineers and Scientists, by Lindholm et.al.</p>
  <p>Chapter 3, page 50.</p>
  <p>Link: <a href="http://smlbook.org/book/sml-book-draft-latest.pdf" rel="nofollow noreferrer">http://smlbook.org/book/sml-book-draft-latest.pdf</a></p>
  <p>Consider the logistic regression for classification.</p>
  <p><span class="math-container">$
  \hat{y}(\mathbf{x})=
  \begin{cases}
      &amp;1   &amp;  g(\mathbf{x})&gt;r \\\\
     &amp;-1   &amp;  g(\mathbf{x})\le r &amp; 
  \end{cases}
  $</span></p>
  <p>The book says &quot;It can be shown that if <span class="math-container">$g(\mathbf{x})=p(y=1|\mathbf{x})$</span>, that is, the model provides
  a correct description of the real-world class probabilities, then the choice <span class="math-container">$ = 0.5$</span> will
  give the smallest possible number of misclassifications on average.&quot;</p>
  <p>I am confused here because if the model provides
  a correct description of the real-world class probabilities, why do we have misclassifications?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-11T18:52:29.310Z
Comments.Text: Could you please give more context for the quote - e.g. chapter name, section name and pages in a specific edition -  because this is probably due to what else is assumed part of the problem (e.g. the quote refers to classifying new data, or allows for g(x) for different examples but same vector to be differen - I think it is simply the latter, that $x_i = x_j$ but $g(x_i) \neq g(x_j)$)
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-11T19:34:40.043Z
Comments.Text: I updated the question to give the book details.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-11T20:07:45.900Z
Comments.Text: Are there two reasons? Non separability of classes and some error due to boundary estimation from finite data?
Comments.UserDisplayName: Jaume Oliver Lafont
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-11T20:10:58.850Z
Comments.Text: I am thinking about a very simple example to illustrate both. Classify numbers into positive and negative from uniform distributions in (-1,0) and (0,1) -separable with estimated boundary not exactly zero but close- and overlapping distributions, for example (-3/4,1/4) and (-1/4, 3/4).
Comments.UserDisplayName: Jaume Oliver Lafont
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-11T21:07:57.693Z
Comments.Text: I think book means something different.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-11T21:11:53.390Z
Comments.Text: I understand your question is "why is the smallest possible number of misclassifications larger than zero".
Comments.UserDisplayName: Jaume Oliver Lafont
Tags: <machine-learning><logistic-regression>
Title: Why can we have misclassifications for a perfect model in logistic regression?
UserDisplayName: DSPinfinity
ThreadText: # Why can we have misclassifications for a perfect model in logistic regression?
  
  (Asked by: username_0 on 2023-11-11)
  
  I am reading the book: MACHINE LEARNING- A First Course for Engineers and Scientists, by Lindholm et.al.
  
  Chapter 3, page 50.
  
  Link: <http://smlbook.org/book/sml-book-draft-latest.pdf>
  
  Consider the logistic regression for classification.
  
  $
  \hat{y}(\mathbf{x})=
  \begin{cases}
  &1 & g(\mathbf{x})>r \\\\
  &-1 & g(\mathbf{x})\le r &
  \end{cases}
  $
  
  The book says "It can be shown that if $g(\mathbf{x})=p(y=1|\mathbf{x})$, that is, the model provides
  a correct description of the real-world class probabilities, then the choice $ = 0.5$ will
  give the smallest possible number of misclassifications on average."
  
  I am confused here because if the model provides
  a correct description of the real-world class probabilities, why do we have misclassifications?
  
  **Tags:** machine-learning, logistic-regression
  > **Comments:**
  > **username_1**: Could you please give more context for the quote - e.g. chapter name, section name and pages in a specific edition -  because this is probably due to what else is assumed part of the problem (e.g. the quote refers to classifying new data, or allows for g(x) for different examples but same vector to be differen - I think it is simply the latter, that $x_i = x_j$ but $g(x_i) \neq g(x_j)$)  
  > **username_0**: I updated the question to give the book details.  
  > **username_2**: Are there two reasons? Non separability of classes and some error due to boundary estimation from finite data?  
  > **username_2**: I am thinking about a very simple example to illustrate both. Classify numbers into positive and negative from uniform distributions in (-1,0) and (0,1) -separable with estimated boundary not exactly zero but close- and overlapping distributions, for example (-3/4,1/4) and (-1/4, 3/4).  
  > **username_0**: I think book means something different.  
  > **username_2**: I understand your question is "why is the smallest possible number of misclassifications larger than zero".  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2023-11-11. Score: 4
  
  Since $g$ is your logistic regression model, over data samples $x$, the output of $g(x)$ is a scalar value between $0$ and $1$ that is usually interpreted as a *probability value*.
  
  * We have that $g(x)=\sigma(W x + b)$ essentially, where $W,b$ are the parameters that you learn and $\sigma$ is the sigmoid squashing function (the one that keeps everything in $[0, 1]$).
  * Furthermore, $g(x)=p(y=1\mid x)$ meaning that $g(x)$ represents the probability that the sample $x$ belongs to the *positive* or, as the book says in page 49, *most probable class* - as a side note, this is something that you can change and usually define before learning.
  
  Now, let's assume that the two classes $y=1$ (the positive or most probable) and $y=0$ (the negative or less probable) have real-world probabilities of: $p(y=1\mid x) = 0.7$ and $p(y=0\mid x) = 1 - p(y=1\mid x) = 0.3$. Let's also be realistic, in the sense that the two class distributions overlap: this implies that you have some misclassification errors due to the fact that is not possible to separate the two classes perfectly. Note: there could be many reasons for this, such as mis-labelling, and similarity of data samples between the two classes.
  
  At this point, we have $g(x)=p(y=1\mid x)$ that is the probability of $x$ to belong to the positive class ($y=1$), implying that there is a $70\%$ chance that $x$ has label $1$. In order to perform a binary classification, you need to get rid of the probabilities by *thresholding them*, such as: $\hat{y}(x) = g(x) > t$.
  
  * If $g(x)>t$ you assign label $1$ (so $\hat{y}(x)=1$), otherwise you assign label $0$.
  * In case the model $g$ predicts the real-world class probabilities correctly (i.e., $p(y=1)=0.7$ and $p(y=0)=0.3$) - as assumed by the book - you have that if the threshold is chosen to be $t=0.5$ (or $r=0.5$ as called in the book), happens that $g(x)>0.5$ minimizes the number of misclassifications, equivalently to maximize the accuracy of predictions.
  * In other words, for a threshold of $0.5$ you get maximum accuracy. But in real classification problems you care also about other measures, such as the AUC of ROC/PR curves, FPR and TPR, F1-score, etc. Basically, you can target a metric and tune the threshold $t$ to get the rate of error that best suits you application.
  
  If still you have doubts about your question:
  
  > if the model provides a correct description of the real-world class probabilities, why do we have misclassifications?
  
  Consider a classifier $g'(x)=0.7$ that always outputs a probability of $0.7$ regardless the data point, $x$. Even if the true class probability is $0.7$, this model is still erroneous $30\%$ of the times when classifying all the $x'$ such that $y(x') = 0$.
  
  
  ---
  
  ### Answer by username_4 on 2023-11-11. Score: -1
  
  Note your book specifically claims 'the model provides a correct description of the real-world class *probabilities*', which exactly means the [multi-explanatory variables logistic function](https://en.wikipedia.org/wiki/Logistic_regression#Many_explanatory_variables,_two_categories) $g(\mathbf{x})$ trained in your logistic binary classifier $\hat{y}(\mathbf{x})$, as a special type of S-shaped sigmoid function transforming the linear combination of input features into a *probability* between 0 and 1 could be possibly correct in the real world. And if this is true, then intuitively $r=0.5$ gives the best overall accuracy and the smallest possible number of misclassifications on average.
  
  In summary it's the logistic model $g(\mathbf{x})$ could possibly provide a correct description of the real-world class probabilities, but even this is the case, the resulting binary classifier $\hat{y}(\mathbf{x})$ would still possibly have misclassification since its accuracy may not be 100% either on your training samples or unseen test samples.
  
  
  ---
  
  