Answers.Body: <p>First assess whether the accuracy is unreasonably high.  You as the domain expert are the best arbiter of that.  Keep in mind that a high accuracy is not always impossible.  For example, 99% accuracy can be seen on MNIST digits dataset.  It is also possible that you have <a href="https://machinelearningmastery.com/data-leakage-machine-learning/" rel="nofollow noreferrer">data leakage</a>, so assess whether this may be at play. The second thing you can do is perform an <a href="https://scikit-learn.org/stable/modules/cross_validation.html" rel="nofollow noreferrer">n-fold cross-validation</a>.  This will provide you with a better estimate (and error bounds) for your model's performance.  If your accuracy remains high on all folds, then you can exclude a spurious split on your holdout validation.</p>
  <p>Another issue is that you claim that you have an imbalanced dataset.  However, your print out shows roughly equal support for 0 and 1 classes.</p>
  <p>Finally, you are performing preprocessing of your features on the total dataset.  Instead, you should apply <code>fit_transform</code> on your training data, and the just apply the <code>transform</code> on your test data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-11T04:25:23.990Z
Answers.Text: I have an imbalanced dataset but the data I show above is just a sample of what my data looks like. Also, why should I should apply fit_transform on my training data, and just apply the transform on my test data?
Answers.UserDisplayName: JayTicku
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-11T04:33:32.083Z
Answers.Text: Ok, makes sense.  In that case, I would say that it is customary to perform evaluation on the total test set, not just a subset that you selected, regardless of whether your dataset is balanced or not.  Regarding the second issue of fit / fit_transform, the suggestion has to do with data leakage.  see [article](https://scikit-learn.org/stable/common_pitfalls.html#data-leakage)
Answers.UserDisplayName: Snehal Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-12T01:10:04.347Z
Answers.Text: I ran a classification report and it returned a score of 0.99 as shown above, so does that say that the model was able to generalize well?
Answers.UserDisplayName: JayTicku
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-10T03:23:59.163Z
Answers.LastActivityDate: 2022-11-10T03:23:59.163Z
Answers.UserDisplayName: Snehal Patel
Body: <p><a href="https://i.sstatic.net/ruz9r.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ruz9r.png" alt="This is how a sample of my data looks like" /></a>
  I am building a model that predicts if a user will like a stock or not based on different features, such as Market Cap, Current Ratio, Sector, Trailing PE, etc. I am going to implement this model in a website, where the model is able to adapt over time to user preferences. I made an example dataset that generally represents how the data that gets passed to the model will look. I have build a KNN model based on that sample dataset, and I get 99% accuracy on the training split and test split. I also get 98%-99% on my classification report even though I have class imbalance, about 500 of one class, and 10500 of another class. My macro average and weighted average are both 99%. I was planning on optimizing this model by hyper tuning, but does anyone know why I got 99% accuracy immediately? This is how a sample of my data looks like above.
  Here is some of my code:</p>
  <pre><code>X = data.drop([&quot;Stocks&quot;, &quot;target&quot;], axis = 1)
  y = data[&quot;target&quot;]
  from sklearn.preprocessing import OneHotEncoder
  from sklearn.compose import ColumnTransformer
  
  categorical_features = [&quot;sector&quot;, &quot;quoteType&quot;]
  one_hot = OneHotEncoder()
  transformer = ColumnTransformer([(&quot;one_hot&quot;,
                                    one_hot,
                                    categorical_features)],
                                 remainder = &quot;passthrough&quot;)
  
  X = transformer.fit_transform(X)
  
  from sklearn.preprocessing import StandardScaler
  scaler = StandardScaler()
  X = scaler.fit_transform(X) 
  X = pd.DataFrame(X)
  from sklearn.model_selection import train_test_split
  from sklearn.neighbors import KNeighborsClassifier
  
  np.random.seed(42)
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
  knn = KNeighborsClassifier(metric = &quot;minkowski&quot;)
  knn.fit(X_train, y_train)
  knn.score(X_test, y_test)
  
  y_preds = knn.predict(X_test)
  
  from sklearn.metrics import classification_report
  print(classification_report(y_test, y_preds))
  
  This produces:
  precision    recall  f1-score   support
  
             0       0.98      1.00      0.99      1227
             1       1.00      0.98      0.99      1012
  
      accuracy                           0.99      2239
     macro avg       0.99      0.99      0.99      2239
  weighted avg       0.99      0.99      0.99      2239
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-09T04:19:23.417Z
Comments.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/140489/discussion-on-question-by-messi10-why-does-knn-model-return-99-accuracy-on-data).
Comments.UserDisplayName: Mithical
Tags: <classification><binary-classification><accuracy><scikit-learn><k-nearest-neighbors>
Title: Why does KNN Model return 99% accuracy on dataset with default parameters?
UserDisplayName: JayTicku
ThreadText: # Why does KNN Model return 99% accuracy on dataset with default parameters?
  
  (Asked by: username_0 on 2022-11-09)
  
  [![This is how a sample of my data looks like](https://i.sstatic.net/ruz9r.png)](https://i.sstatic.net/ruz9r.png)
  I am building a model that predicts if a user will like a stock or not based on different features, such as Market Cap, Current Ratio, Sector, Trailing PE, etc. I am going to implement this model in a website, where the model is able to adapt over time to user preferences. I made an example dataset that generally represents how the data that gets passed to the model will look. I have build a KNN model based on that sample dataset, and I get 99% accuracy on the training split and test split. I also get 98%-99% on my classification report even though I have class imbalance, about 500 of one class, and 10500 of another class. My macro average and weighted average are both 99%. I was planning on optimizing this model by hyper tuning, but does anyone know why I got 99% accuracy immediately? This is how a sample of my data looks like above.
  Here is some of my code:
  
  ```
  X = data.drop(["Stocks", "target"], axis = 1)
  y = data["target"]
  from sklearn.preprocessing import OneHotEncoder
  from sklearn.compose import ColumnTransformer
  
  categorical_features = ["sector", "quoteType"]
  one_hot = OneHotEncoder()
  transformer = ColumnTransformer([("one_hot",
                                    one_hot,
                                    categorical_features)],
                                 remainder = "passthrough")
  
  X = transformer.fit_transform(X)
  
  from sklearn.preprocessing import StandardScaler
  scaler = StandardScaler()
  X = scaler.fit_transform(X) 
  X = pd.DataFrame(X)
  from sklearn.model_selection import train_test_split
  from sklearn.neighbors import KNeighborsClassifier
  
  np.random.seed(42)
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
  knn = KNeighborsClassifier(metric = "minkowski")
  knn.fit(X_train, y_train)
  knn.score(X_test, y_test)
  
  y_preds = knn.predict(X_test)
  
  from sklearn.metrics import classification_report
  print(classification_report(y_test, y_preds))
  
  This produces:
  precision    recall  f1-score   support
  
             0       0.98      1.00      0.99      1227
             1       1.00      0.98      0.99      1012
  
      accuracy                           0.99      2239
     macro avg       0.99      0.99      0.99      2239
  weighted avg       0.99      0.99      0.99      2239
  
  ```
  
  **Tags:** classification, binary-classification, accuracy, scikit-learn, k-nearest-neighbors
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/140489/discussion-on-question-by-messi10-why-does-knn-model-return-99-accuracy-on-data).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-11-10. Score: 1 | Accepted answer
  
  First assess whether the accuracy is unreasonably high. You as the domain expert are the best arbiter of that. Keep in mind that a high accuracy is not always impossible. For example, 99% accuracy can be seen on MNIST digits dataset. It is also possible that you have [data leakage](https://machinelearningmastery.com/data-leakage-machine-learning/), so assess whether this may be at play. The second thing you can do is perform an [n-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html). This will provide you with a better estimate (and error bounds) for your model's performance. If your accuracy remains high on all folds, then you can exclude a spurious split on your holdout validation.
  
  Another issue is that you claim that you have an imbalanced dataset. However, your print out shows roughly equal support for 0 and 1 classes.
  
  Finally, you are performing preprocessing of your features on the total dataset. Instead, you should apply `fit_transform` on your training data, and the just apply the `transform` on your test data.
  
  > **Comments:**
  > **username_0**: I have an imbalanced dataset but the data I show above is just a sample of what my data looks like. Also, why should I should apply fit_transform on my training data, and just apply the transform on my test data?  
  > **username_2**: Ok, makes sense.  In that case, I would say that it is customary to perform evaluation on the total test set, not just a subset that you selected, regardless of whether your dataset is balanced or not.  Regarding the second issue of fit / fit_transform, the suggestion has to do with data leakage.  see [article](https://scikit-learn.org/stable/common_pitfalls.html#data-leakage)  
  > **username_0**: I ran a classification report and it returned a score of 0.99 as shown above, so does that say that the model was able to generalize well?  
  
  ---
  
  