Body: <p>I’ve been experimenting on several datasets and found something very strange while implementing ML.<br>
  I’ll Explain after the code…</p>
  
  <pre><code>import numpy as np
  from sklearn import datasets
  iris = datasets.load_iris()
  
  # 4 features in np array - 150 rows
  
  case = 1        # change cases to see variation
  
  if case == 1:   # first feature deleted
      iris.data = np.delete(iris.data,0, 1)
  
  if case == 2:   # first 2 features deleted
      iris.data = np.delete(iris.data,0, 1)
      iris.data = np.delete(iris.data,0, 1)
  
  if case == 3:   # first 3 features deleted (1 feature left)
      iris.data = np.delete(iris.data,0, 1)
      iris.data = np.delete(iris.data,0, 1)
      iris.data = np.delete(iris.data,0, 1)
  
  if case == 4:   # only second feature deleted from np array
      iris.data = np.delete(iris.data,1, 1)
  
  if case == 5:   # only third feature deleted from np array
      iris.data = np.delete(iris.data,2, 1)
  
  if case == 6:   # only last feature deleted from np array
      iris.data = np.delete(iris.data,3, 1)
  
  # print iris.data
  # exit()
  
  from sklearn.naive_bayes import GaussianNB
  gnb = GaussianNB()
  pred = gnb.fit(iris.data, iris.target).predict(iris.data)
  # pred = gnb.fit(iris.data, iris.target).predict(test_data)
  
  from sklearn.metrics import accuracy_score
  print accuracy_score(iris.target, pred)
  </code></pre>
  
  <p>I’m using the basic fisher iris dataset from sklearn, it has 150 rows and 4 columns (features).<br>
  Using training data as test data.</p>
  
  <p>So i tried to remove a few features and see if accuracy changed.
  And I thought it would.<br>
  But till case <strong>1</strong>, <strong>2</strong>, and <strong>3</strong>, I removed 1, 2 and 3 features respectively and there was no change in the accuracy. It stayed 96%.</p>
  
  <p>Then on running cases <strong>4</strong>,<strong>5</strong> and <strong>6</strong>, The accuracy changed. Why?</p>
  
  <p>On comparing case <strong>2</strong> and <strong>4</strong>,<br>
  Both have second feature removed from the dataset, so clearly, removing second feature is responsible for change in accuracy (as seen in case <strong>4</strong>)<br>
  So why does it not change in case <strong>2</strong>?<br>
  Just because it had first feature removed too? to balance out the second? (If that were true, case <strong>1</strong> would have given different accuracy)<br>
  Why does accuracy not change in first 3 cases, but it does in the last 3 cases?</p>
  
  <p>Is ML dependent on the order in which features are feeded to the algorithm?</p>
  
  <p>What am I missing here?</p>
  
  <p>Would be great if someone could clear this doubt.</p>
  
  <p>Thanks!</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-19T06:55:03.983Z
Comments.Text: Welcome to AI stack-exchange. Good question by the way :). I just wanted to point out to that "Using training data as test data." - is a recipe for overfitting. Which means that the accuracy you are seeing is not the real accuracy you can expect your model to get if you test on a dataset your model hasn't seen yet.With that said, I understand why you did that.... the iris is a small dataset :(
Comments.UserDisplayName: Tshilidzi Mudau
Tags: <machine-learning>
Title: Is accuracy of a machine learning algorithm independent of the features?
UserDisplayName: Aneesh Kumar
ThreadText: # Is accuracy of a machine learning algorithm independent of the features?
  
  (Asked by: username_0 on 2017-10-19)
  
  I’ve been experimenting on several datasets and found something very strange while implementing ML.
  I’ll Explain after the code…
  import numpy as np
  from sklearn import datasets
  iris = datasets.load_iris()
  
  # 4 features in np array - 150 rows
  
  case = 1        # change cases to see variation
  
  if case == 1:   # first feature deleted
      iris.data = np.delete(iris.data,0, 1)
  
  if case == 2:   # first 2 features deleted
      iris.data = np.delete(iris.data,0, 1)
      iris.data = np.delete(iris.data,0, 1)
  
  if case == 3:   # first 3 features deleted (1 feature left)
      iris.data = np.delete(iris.data,0, 1)
      iris.data = np.delete(iris.data,0, 1)
      iris.data = np.delete(iris.data,0, 1)
  
  if case == 4:   # only second feature deleted from np array
      iris.data = np.delete(iris.data,1, 1)
  
  if case == 5:   # only third feature deleted from np array
      iris.data = np.delete(iris.data,2, 1)
  
  if case == 6:   # only last feature deleted from np array
      iris.data = np.delete(iris.data,3, 1)
  
  # print iris.data
  # exit()
  
  from sklearn.naive_bayes import GaussianNB
  gnb = GaussianNB()
  pred = gnb.fit(iris.data, iris.target).predict(iris.data)
  # pred = gnb.fit(iris.data, iris.target).predict(test_data)
  
  from sklearn.metrics import accuracy_score
  print accuracy_score(iris.target, pred)
  
  I’m using the basic fisher iris dataset from sklearn, it has 150 rows and 4 columns (features).
  Using training data as test data.
  So i tried to remove a few features and see if accuracy changed.
  And I thought it would.
  But till case 1, 2, and 3, I removed 1, 2 and 3 features respectively and there was no change in the accuracy. It stayed 96%.
  Then on running cases 4,5 and 6, The accuracy changed. Why?
  On comparing case 2 and 4,
  Both have second feature removed from the dataset, so clearly, removing second feature is responsible for change in accuracy (as seen in case 4)
  So why does it not change in case 2?
  Just because it had first feature removed too? to balance out the second? (If that were true, case 1 would have given different accuracy)
  Why does accuracy not change in first 3 cases, but it does in the last 3 cases?
  Is ML dependent on the order in which features are feeded to the algorithm?
  What am I missing here?
  Would be great if someone could clear this doubt.
  Thanks!
  
  **Tags:** machine-learning
  > **Comments:**
  > **username_1**: Welcome to AI stack-exchange. Good question by the way :). I just wanted to point out to that "Using training data as test data." - is a recipe for overfitting. Which means that the accuracy you are seeing is not the real accuracy you can expect your model to get if you test on a dataset your model hasn't seen yet.With that said, I understand why you did that.... the iris is a small dataset :(  
  
  ---
  
  ## 0 answers
  
  