Answers.Body: <p>You provided very little context here without explicit reference, though it looks like the topic here is instance based machine learning such as KNN method using potentially beneficial scaled Euclidean distance. You need to first know why the left graph case of feature-dependent scaling is effective compared to its original unstretched classification learning in terms of their decision boundaries.</p>
  <p>Consider a hypothetical dataset with two classes (spam and non-spam) having two features (length and ratio of number of blacklisted keywords). Obviously the original features have wildly different scales thus KNN method would provide an implicit highly nonlinear irregular decision boundary by the meshgrid in below plot around the possibly overlapping region of the feature space (code is provided at the end).<a href="https://i.sstatic.net/kQBBU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kQBBU.png" alt="enter image description here" /></a></p>
  <p>But if you scale the feature of ratio of blacklisted keywords to be of similar range of the other email length feature, the below implicit decision boundary provided by the same KNN classifier becomes much more regular and thus effective in many respects such as variance reduction.<a href="https://i.sstatic.net/RxjGc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RxjGc.png" alt="enter image description here" /></a></p>
  <p>This is exactly the case in your left graph where stretching the feature <span class="math-container">$x_2$</span> makes the classification boundary much more regular and statistically effective to predict any new data.</p>
  <p>But if you're in the middle class-dependent scaling case, any such scaling elongation or compression obviously aren't effective compared to their original unscaled graph since any such scaling doesn't make the bottom-left possibly overlapping corner of the two classes any more clearly separable and regular. Similar case goes for the right case of correlated features since the usual linear correlation between <span class="math-container">$x_1$</span> and <span class="math-container">$x_2$</span> won't get affected at all by any scaling.</p>
  <pre><code>import numpy as np
  import matplotlib.pyplot as plt
  from sklearn.neighbors import KNeighborsClassifier
  from matplotlib.colors import ListedColormap
  
  np.random.seed(42)
  
  spam_length = np.random.uniform(150, 500, 50)
  spam_blacklisted_keyword = np.random.uniform(0.2, 0.8, 50)
  spams = np.column_stack((spam_length, spam_blacklisted_keyword))
  
  nonspam_length = np.random.uniform(30, 300, 50)
  nonspam_blacklisted_keyword = np.random.uniform(0, 0.3, 50)
  nonspams = np.column_stack((nonspam_length, nonspam_blacklisted_keyword))
  
  # spam_length = np.random.uniform(150, 500, 50)
  # spam_blacklisted_keyword = np.random.uniform(0.2*50, 0.8*50, 50)
  # spams = np.column_stack((spam_length, spam_blacklisted_keyword))
  
  # nonspam_length = np.random.uniform(30, 300, 50)
  # nonspam_blacklisted_keyword = np.random.uniform(0*50, 0.3*50, 50)
  # nonspams = np.column_stack((nonspam_length, nonspam_blacklisted_keyword))
  
  X = np.vstack((spams, nonspams))
  y = np.array([0] * 50 + [1] * 50)
  
  knn = KNeighborsClassifier(n_neighbors=5)
  knn.fit(X, y)
  
  h = .2  # Step size in the mesh
  x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
  y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
  xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
  Z = knn.predict(np.c_[xx.ravel(), yy.ravel()])
  
  # Plotting the decision boundary
  Z = Z.reshape(xx.shape)
  plt.contourf(xx, yy, Z, alpha=0.4)
  plt.scatter(spams[:, 0], spams[:, 1], c='red', label='spams')
  plt.scatter(nonspams[:, 0], nonspams[:, 1], c='green', label='nonspams')
  plt.xlabel('Length')
  plt.ylabel('# of blacklisted keywords')
  plt.ylim(0, 1)
  plt.legend()
  plt.title('KNN Decision Boundary')
  plt.show()
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-13T21:42:27.867Z
Answers.LastActivityDate: 2023-11-18T00:17:31.107Z
Answers.LastEditDate: 2023-11-18T00:17:31.107Z
Answers.UserDisplayName: cinch
Body: <p>The following is from a machine learning book.</p>
  <p><a href="https://i.sstatic.net/oZpTT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/oZpTT.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/NqyLc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NqyLc.png" alt="enter image description here" /></a></p>
  <p>I did not understand the explanation given in the figure caption. Could some expert make it clear? Why is the stretching class-dependent for the center case?</p>
  
Tags: <machine-learning><classification><pattern-recognition>
Title: Unclear points in scaled Euclidean distance
UserDisplayName: DSPinfinity
ThreadText: # Unclear points in scaled Euclidean distance
  
  (Asked by: username_0 on 2023-10-17)
  
  The following is from a machine learning book.
  
  [![enter image description here](https://i.sstatic.net/oZpTT.png)](https://i.sstatic.net/oZpTT.png)
  
  [![enter image description here](https://i.sstatic.net/NqyLc.png)](https://i.sstatic.net/NqyLc.png)
  
  I did not understand the explanation given in the figure caption. Could some expert make it clear? Why is the stretching class-dependent for the center case?
  
  **Tags:** machine-learning, classification, pattern-recognition
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-13. Score: 0
  
  You provided very little context here without explicit reference, though it looks like the topic here is instance based machine learning such as KNN method using potentially beneficial scaled Euclidean distance. You need to first know why the left graph case of feature-dependent scaling is effective compared to its original unstretched classification learning in terms of their decision boundaries.
  Consider a hypothetical dataset with two classes (spam and non-spam) having two features (length and ratio of number of blacklisted keywords). Obviously the original features have wildly different scales thus KNN method would provide an implicit highly nonlinear irregular decision boundary by the meshgrid in below plot around the possibly overlapping region of the feature space (code is provided at the end).
  But if you scale the feature of ratio of blacklisted keywords to be of similar range of the other email length feature, the below implicit decision boundary provided by the same KNN classifier becomes much more regular and thus effective in many respects such as variance reduction.
  This is exactly the case in your left graph where stretching the feature $x_2$ makes the classification boundary much more regular and statistically effective to predict any new data.
  But if you're in the middle class-dependent scaling case, any such scaling elongation or compression obviously aren't effective compared to their original unscaled graph since any such scaling doesn't make the bottom-left possibly overlapping corner of the two classes any more clearly separable and regular. Similar case goes for the right case of correlated features since the usual linear correlation between $x_1$ and $x_2$ won't get affected at all by any scaling.
  import numpy as np
  import matplotlib.pyplot as plt
  from sklearn.neighbors import KNeighborsClassifier
  from matplotlib.colors import ListedColormap
  
  np.random.seed(42)
  
  spam_length = np.random.uniform(150, 500, 50)
  spam_blacklisted_keyword = np.random.uniform(0.2, 0.8, 50)
  spams = np.column_stack((spam_length, spam_blacklisted_keyword))
  
  nonspam_length = np.random.uniform(30, 300, 50)
  nonspam_blacklisted_keyword = np.random.uniform(0, 0.3, 50)
  nonspams = np.column_stack((nonspam_length, nonspam_blacklisted_keyword))
  
  # spam_length = np.random.uniform(150, 500, 50)
  # spam_blacklisted_keyword = np.random.uniform(0.2*50, 0.8*50, 50)
  # spams = np.column_stack((spam_length, spam_blacklisted_keyword))
  
  # nonspam_length = np.random.uniform(30, 300, 50)
  # nonspam_blacklisted_keyword = np.random.uniform(0*50, 0.3*50, 50)
  # nonspams = np.column_stack((nonspam_length, nonspam_blacklisted_keyword))
  
  X = np.vstack((spams, nonspams))
  y = np.array([0] * 50 + [1] * 50)
  
  knn = KNeighborsClassifier(n_neighbors=5)
  knn.fit(X, y)
  
  h = .2  # Step size in the mesh
  x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
  y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
  xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
  Z = knn.predict(np.c_[xx.ravel(), yy.ravel()])
  
  # Plotting the decision boundary
  Z = Z.reshape(xx.shape)
  plt.contourf(xx, yy, Z, alpha=0.4)
  plt.scatter(spams[:, 0], spams[:, 1], c='red', label='spams')
  plt.scatter(nonspams[:, 0], nonspams[:, 1], c='green', label='nonspams')
  plt.xlabel('Length')
  plt.ylabel('# of blacklisted keywords')
  plt.ylim(0, 1)
  plt.legend()
  plt.title('KNN Decision Boundary')
  plt.show()
  
  
  ---
  
  