Higher accuracy in training set than test set in KNN classifier If I got an accuracy of 95% in the training set, and 80% accuracy in the test set, what could be the explanation for that? it sounds like a pretty basic question, but I just can't put it into words correctly.
It's supposed to be like that, I know that far. I interpret it like this:
" There is overfitting in the KNN model, the model learned the training data too well, and now it uses the features it learned in the training data in a false way. The model is familiar with the training data more, after all, it was trained on it, and the test data it has never seen before so it's only natural to get these results ".
What do you think of this explanation?

If you use too small values for k (e.g. k=1) you get decision boundaries which are very erratic and that leads to overfitting.
It means that if k is too small the decision boundaries depend highly on the sample that you have. If you had another sample (by chance) the decision boundaries would look very different.
Below are two pictures from the book "The Elements of Statistical Learning Data Mining, Inference, and Prediction" by Hastie, Tibshirani and Friedman illustrating this. Choosing larger k leads to more smooth decision boundaries and less variance in the model.
Whether a gap of 95% to 80% is reasonable depends on the sample size and the dimensionality of your data set I would say. In general though the gap will widen for smaller k and get smaller for larger k, but never disappear.


