Answers.Body: <p>For noisy random test dataset <span class="math-container">$(\mathbb{x}, y)$</span>, the similarity-based non-optimal nearest neighbor classifier <span class="math-container">$g_N$</span> will have out-of-sample misclassification error probability as expressed by <span class="math-container">$P[g_N(\mathbb{x}) \neq y]$</span>, and obviously if its nearest neighbor in the training data is erroneously labeled then error probability is pretty high for KNN (=1 for 1NN) classifier, thus nearest neighbor classifier makes a mistake if there's an error on the test point <em>or</em> on its nearest neighbor.</p>
  <p>On the other hand, since your assumed optimal (Bayes) classifier <span class="math-container">$f$</span> for binary classification where output, say <span class="math-container">$1$</span> and <span class="math-container">$-1$</span>, is modelled noisy via a unique conditional probability distribution <span class="math-container">$\pi(\mathbb{x})$</span>, for the same noisy random test dataset <span class="math-container">$(\mathbb{x}, y)$</span> its out-of-sample misclassification error probability is expressed by <span class="math-container">$P[f(\mathbb{x}) \neq y]=\text{min}(\pi(\mathbb{x}), 1−\pi(\mathbb{x}))$</span>. Note the erroneously labeled training data outlier(s) would not have much effect on the assumed optimal conditional distribution <span class="math-container">$\pi(\mathbb{x})$</span> as long as the whole training data is enough and representative for its population</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T01:22:31.393Z
Answers.Text: you say " ...there's an error on the test point or on its nearest neighbor." What does "there's an error on the test point " mean?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T02:36:41.673Z
Answers.Text: It simply means for some test data features $\mathbb{x}$ its $y$ class label is erroneous, for instance, -1 instead of 1, which is possible.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T08:30:05.150Z
Answers.Text: I am not sure. To measure error on test point $\mathbf{}x$, you have to know its ground true, ie, the true class label. Otherwise, you cannot define the error.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T16:23:16.830Z
Answers.Text: Error in a test data is in the exactly same sense as error in a training data which as explained in my answer could cause a KNN classifier to misclassify even a correctly labeled test data, once you understand there may be error in training data, you understand same in test data, as both are sometimes partitioned from a single dataset via (k-fold) cross validation. The error measured is for the classifier we're pursuing, not the so called ground truth training/test data themselves which we cannot learn but accept albeit may be erroneous, 'ground truth' naming seems misleading you.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T17:54:13.610Z
Answers.Text: Sorry, I said wrongly. You measure the error of a classifier with respect to class labels (whether the class labels are correct or wrong). Hence, I still do not understand the term "error on a test point". You do not define error based on ground truths (not based on the "correct" labels of the data points).
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T19:19:55.673Z
Answers.Text: Even the optimal classifier would make a mistake if there's an error on the test point whose probability is expressed in the 2nd paragraph of my above answer, the 'error' here is defined to be the classifier's possible out-of-sample misclassification for a randomly drawn sample out of the test dataset $(\mathbb{x}, y)$ which is a random vector, thus it's not defined based on any ground truths but based on the said randomly drawn test data sample. Your mentioned ground truth is only a realized drawn sample from an inherently random source. Hope this clarifies more.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-22T00:31:32.023Z
Answers.Text: Thanks for your efforts but I am very hardly convinced from your explanations.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T08:00:21.747Z
Answers.Text: Take some time to do the exercise and understand why above out-of-sample misclassification error probability $P[f(\mathbb{x}) \neq y]=\text{min}(\pi(\mathbb{x}), 1−\pi(\mathbb{x}))$ holds for the optimal classifier $f$. Once you understand this result, then you know the whole population containing unseen test data's class label $y$ is implicitly assumed to be a random variable whose probability is conditioned on the test data feature $\mathbb{x}$, and this is the source of the unavoidable noisy data either in training or test dataset for any optimal (Bayes) classifier. Data is always noisy.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T01:15:36.133Z
Answers.LastActivityDate: 2023-11-21T01:15:36.133Z
Answers.UserDisplayName: cinch
Body: <p>Let f(x) be optimal classifier for binary classification where output is modelled noisy.</p>
  <p>What does it mean &quot;f(x) makes a mistake only if there is an error on the test point x&quot;? Basically,
  what is meant by &quot;error on the test point x&quot;?</p>
  <p>For content, please see:</p>
  <p><a href="https://amlbook.com/eChapters/6-Oct2022-readeronly.pdf" rel="nofollow noreferrer">https://amlbook.com/eChapters/6-Oct2022-readeronly.pdf</a></p>
  <p>Password:Paraskavedekatriaphobia</p>
  <p>Page: 6, last paragraph</p>
  
Tags: <machine-learning><classification><k-nearest-neighbors>
Title: Meaning of "error on the test point x" in optimal classifier for binary classification
UserDisplayName: DSPinfinity
ThreadText: # Meaning of "error on the test point x" in optimal classifier for binary classification
  
  (Asked by: username_0 on 2023-11-20)
  
  Let f(x) be optimal classifier for binary classification where output is modelled noisy.
  
  What does it mean "f(x) makes a mistake only if there is an error on the test point x"? Basically,
  what is meant by "error on the test point x"?
  
  For content, please see:
  
  <https://amlbook.com/eChapters/6-Oct2022-readeronly.pdf>
  
  Password:Paraskavedekatriaphobia
  
  Page: 6, last paragraph
  
  **Tags:** machine-learning, classification, k-nearest-neighbors
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-21. Score: 0
  
  For noisy random test dataset $(\mathbb{x}, y)$, the similarity-based non-optimal nearest neighbor classifier $g\_N$ will have out-of-sample misclassification error probability as expressed by $P[g\_N(\mathbb{x}) \neq y]$, and obviously if its nearest neighbor in the training data is erroneously labeled then error probability is pretty high for KNN (=1 for 1NN) classifier, thus nearest neighbor classifier makes a mistake if there's an error on the test point *or* on its nearest neighbor.
  
  On the other hand, since your assumed optimal (Bayes) classifier $f$ for binary classification where output, say $1$ and $-1$, is modelled noisy via a unique conditional probability distribution $\pi(\mathbb{x})$, for the same noisy random test dataset $(\mathbb{x}, y)$ its out-of-sample misclassification error probability is expressed by $P[f(\mathbb{x}) \neq y]=\text{min}(\pi(\mathbb{x}), 1−\pi(\mathbb{x}))$. Note the erroneously labeled training data outlier(s) would not have much effect on the assumed optimal conditional distribution $\pi(\mathbb{x})$ as long as the whole training data is enough and representative for its population
  
  > **Comments:**
  > **username_0**: you say " ...there's an error on the test point or on its nearest neighbor." What does "there's an error on the test point " mean?  
  > **username_1**: It simply means for some test data features $\mathbb{x}$ its $y$ class label is erroneous, for instance, -1 instead of 1, which is possible.  
  > **username_0**: I am not sure. To measure error on test point $\mathbf{}x$, you have to know its ground true, ie, the true class label. Otherwise, you cannot define the error.  
  > **username_1**: Error in a test data is in the exactly same sense as error in a training data which as explained in my answer could cause a KNN classifier to misclassify even a correctly labeled test data, once you understand there may be error in training data, you understand same in test data, as both are sometimes partitioned from a single dataset via (k-fold) cross validation. The error measured is for the classifier we're pursuing, not the so called ground truth training/test data themselves which we cannot learn but accept albeit may be erroneous, 'ground truth' naming seems misleading you.  
  > **username_0**: Sorry, I said wrongly. You measure the error of a classifier with respect to class labels (whether the class labels are correct or wrong). Hence, I still do not understand the term "error on a test point". You do not define error based on ground truths (not based on the "correct" labels of the data points).  
  > **username_1**: Even the optimal classifier would make a mistake if there's an error on the test point whose probability is expressed in the 2nd paragraph of my above answer, the 'error' here is defined to be the classifier's possible out-of-sample misclassification for a randomly drawn sample out of the test dataset $(\mathbb{x}, y)$ which is a random vector, thus it's not defined based on any ground truths but based on the said randomly drawn test data sample. Your mentioned ground truth is only a realized drawn sample from an inherently random source. Hope this clarifies more.  
  > **username_0**: Thanks for your efforts but I am very hardly convinced from your explanations.  
  > **username_1**: Take some time to do the exercise and understand why above out-of-sample misclassification error probability $P[f(\mathbb{x}) \neq y]=\text{min}(\pi(\mathbb{x}), 1−\pi(\mathbb{x}))$ holds for the optimal classifier $f$. Once you understand this result, then you know the whole population containing unseen test data's class label $y$ is implicitly assumed to be a random variable whose probability is conditioned on the test data feature $\mathbb{x}$, and this is the source of the unavoidable noisy data either in training or test dataset for any optimal (Bayes) classifier. Data is always noisy.  
  
  ---
  
  