Body: <p>I am trying to perform binary classification of search results based on the relevance to the query.
  I followed this <a href="https://towardsdatascience.com/support-vector-machine-introduction-to-machine-learning-algorithms-934a444fca47" rel="nofollow noreferrer">tutorial</a> on how to make an SVM, and I got it to work with a small iris dataset. Now, I am attempting to use the LETOR 4.0 MQ2007 dataset by Microsoft to classify. The dataset has 21 input vectors as well as a score from 0 to 2. I classified 0 as -1 and 1, 2 as 1. My algorithm reaches 57.4% accuracy after 1000 epochs with 500 samples of each classification. My learning rate is 0.0001. here is my code.</p>
  
  <pre><code>from tqdm import tqdm
  import numpy as np
  from sklearn.metrics import accuracy_score
  
  
  print("-------------------------------------")
  choice = input("Train or Test: ")
  print("-------------------------------------")
  
  # HYPERPARAMETERS
  feature_num = 21
  epochs = 1000
  sample_size = 500
  learning_rate = 0.0001
  
  if choice == "Train":
  
      out_file = open('weights.txt', 'w')
      out_file.close()
  
      print("Serializing Train Data...")
  
      # SERIALIZE DATA
      file = open('train.txt')
      train_set = file.read().splitlines()
      positive = []
      negative = []
  
      # GRAB TRAINING SAMPLES
      for i in train_set:
          if (i[0] == '1' or i[0] == '2') and len(positive) &lt; sample_size:
              positive.append(i)
          if (i[0] == '0') and len(negative) &lt; sample_size:
              negative.append(i)
  
      train_set = positive+negative
      file.close()
  
      features = []
      query = []
  
      # CREATE TRAINING VECTORS
      alpha = np.full(feature_num, learning_rate)
      weights = np.zeros((len(train_set), feature_num))
      output = np.zeros((len(train_set), feature_num))
      score = np.zeros((len(train_set), feature_num))
  
      for i in tqdm(range(len(train_set))):
          elements = train_set[i].split(' ')
          if int(elements[0]) == 0:
              score[i] = [-1] * feature_num
          else:
              score[i] = [1] * feature_num
  
          query.append(int(elements[1].split(':')[1]))
          tmp = []
          for feature in elements[2:2+feature_num]:
              if feature.split(':')[1] == 'NULL':
                  tmp.append(0.0)
              else:
                  tmp.append(float(feature.split(':')[1]))
          features.append(tmp)
  
      features = np.asarray(features)
  
      print("-------------------------------------")
      print("Training Initialized...")
  
      # TRAIN MODEL
      for i in tqdm(range(epochs)):
  
          # FORWARD y = sum(wx)
          for sample in range(len(train_set)):
              output[sample] = weights[sample]*features[sample]
              output[sample] = np.full((feature_num), np.sum(output[sample]))
  
          # NORMALIZE NEGATIVE SIGNS
          output = output*score
          # UPDATE WEIGHTS
          count = 0
          for val in output:
              if(val[0] &gt;= 1):
                  cost = 0
                  weights = weights - alpha * (2 * 1/epochs * weights)
              else:
                  cost = 1 - val[0]
                  # WEIGHTS = WEIGHTS + LEARNING RATE * [X] * [Y]
                  weights = weights + alpha * (features[count] * score[count] - 2 * 1/epochs * weights)
  
              count += 1
  
      # EXPORT WEIGHTS
      out_file = open('weights.txt', 'a+')
      for i in weights[0]:
          out_file.write(str(i)+'\n')
      out_file.close()
  
  elif choice == "Test":
  
      print("Serializing Test Data...")
  
      # SERIALIZE DATA
      file = open('train.txt')
      train_set = file.read().splitlines()
      positive = []
      negative = []
      for i in train_set:
          if (i[0] == '1' or i[0] == '2') and len(positive) &lt; sample_size:
              positive.append(i)
          if (i[0] == '0') and len(negative) &lt; sample_size:
              negative.append(i)
  
      test_set = positive+negative
  
      file = open('weights.txt', 'r').read().splitlines()
      weights = np.zeros((len(test_set), feature_num))
  
      # CREATE TEST SET
      for i in range(len(weights)):
          weights[i] = file
      features = []
      query = []
      output = np.zeros((len(test_set), feature_num))
      score = np.zeros((len(test_set)))
  
      for i in tqdm(range(len(test_set))):
          elements = test_set[i].split(' ')
          if int(elements[0]) == 0:
              score[i] = -1
          else:
              score[i] = 1
  
          query.append(int(elements[1].split(':')[1]))
          tmp = []
          for feature in elements[2:2+feature_num]:
              if feature.split(':')[1] == 'NULL':
                  tmp.append(0.0)
              else:
                  tmp.append(float(feature.split(':')[1]))
          features.append(tmp)
  
      features = np.asarray(features)
  
      for sample in range(len(test_set)):
          output[sample] = weights[sample]*features[sample]
          output[sample] = np.full((feature_num), np.sum(output[sample]))
  
      predictions = []
      for val in output:
          if(val[0] &gt; 1):
              predictions.append(1)
          else:
              predictions.append(-1)
  
      print("-------------------------------------")
      print("Predicting...")
      print("-------------------------------------")
      print("Prediction finished with "+str(accuracy_score(score, predictions)*100)+"% accuracy.")
  
  </code></pre>
  
  <p>My training algorithm</p>
  
  <pre><code>if(val[0] &gt;= 1):
      cost = 0
      weights = weights - alpha * (2 * 1/epochs * weights)
  else:
      cost = 1 - val[0]
      # WEIGHTS = WEIGHTS + LEARNING RATE * [X] * [Y]
      weights = weights + alpha * (features[count] * score[count] - 2 * 1/epochs * weights)
  
  </code></pre>
  
  <p>What could I do to help the model train? Am I not giving it enough time? Is the algorithm wrong? Are the hyperparameters ok?
  Thanks for all your help.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-07T01:22:09.543Z
Comments.Text: Can you clarify this part "I classified 0 as -1 and 1, 2 as 1"? Does it mean that the original datasets has 3 classes and you're joining two classes so that you can perform binary classification? Also, you have only a total of 1k training examples? That's a relatively small dataset.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-07T03:21:37.610Z
Comments.Text: Yes. The samples are scored 0 - 2, so I classified 0 as -1, and 1 and 2 as 1. Is this not a good way to do it? What would be an adequate sample size? Is my training algorithm ok?
Comments.UserDisplayName: iamPres
Tags: <ai-design><classification><support-vector-machine>
Title: Why is my SVM not reaching good accuracy when trained to perform binary classification of search results?
UserDisplayName: iamPres
ThreadText: # Why is my SVM not reaching good accuracy when trained to perform binary classification of search results?
  
  (Asked by: username_0 on 2020-04-06)
  
  I am trying to perform binary classification of search results based on the relevance to the query.
  I followed this tutorial on how to make an SVM, and I got it to work with a small iris dataset. Now, I am attempting to use the LETOR 4.0 MQ2007 dataset by Microsoft to classify. The dataset has 21 input vectors as well as a score from 0 to 2. I classified 0 as -1 and 1, 2 as 1. My algorithm reaches 57.4% accuracy after 1000 epochs with 500 samples of each classification. My learning rate is 0.0001. here is my code.
  from tqdm import tqdm
  import numpy as np
  from sklearn.metrics import accuracy_score
  
  
  print("-------------------------------------")
  choice = input("Train or Test: ")
  print("-------------------------------------")
  
  # HYPERPARAMETERS
  feature_num = 21
  epochs = 1000
  sample_size = 500
  learning_rate = 0.0001
  
  if choice == "Train":
  
      out_file = open('weights.txt', 'w')
      out_file.close()
  
      print("Serializing Train Data...")
  
      # SERIALIZE DATA
      file = open('train.txt')
      train_set = file.read().splitlines()
      positive = []
      negative = []
  
      # GRAB TRAINING SAMPLES
      for i in train_set:
          if (i[0] == '1' or i[0] == '2') and len(positive) < sample_size:
              positive.append(i)
          if (i[0] == '0') and len(negative) < sample_size:
              negative.append(i)
  
      train_set = positive+negative
      file.close()
  
      features = []
      query = []
  
      # CREATE TRAINING VECTORS
      alpha = np.full(feature_num, learning_rate)
      weights = np.zeros((len(train_set), feature_num))
      output = np.zeros((len(train_set), feature_num))
      score = np.zeros((len(train_set), feature_num))
  
      for i in tqdm(range(len(train_set))):
          elements = train_set[i].split(' ')
          if int(elements[0]) == 0:
              score[i] = [-1] * feature_num
          else:
              score[i] = [1] * feature_num
  
          query.append(int(elements[1].split(':')[1]))
          tmp = []
          for feature in elements[2:2+feature_num]:
              if feature.split(':')[1] == 'NULL':
                  tmp.append(0.0)
              else:
                  tmp.append(float(feature.split(':')[1]))
          features.append(tmp)
  
      features = np.asarray(features)
  
      print("-------------------------------------")
      print("Training Initialized...")
  
      # TRAIN MODEL
      for i in tqdm(range(epochs)):
  
          # FORWARD y = sum(wx)
          for sample in range(len(train_set)):
              output[sample] = weights[sample]*features[sample]
              output[sample] = np.full((feature_num), np.sum(output[sample]))
  
          # NORMALIZE NEGATIVE SIGNS
          output = output*score
          # UPDATE WEIGHTS
          count = 0
          for val in output:
              if(val[0] >= 1):
                  cost = 0
                  weights = weights - alpha * (2 * 1/epochs * weights)
              else:
                  cost = 1 - val[0]
                  # WEIGHTS = WEIGHTS + LEARNING RATE * [X] * [Y]
                  weights = weights + alpha * (features[count] * score[count] - 2 * 1/epochs * weights)
  
              count += 1
  
      # EXPORT WEIGHTS
      out_file = open('weights.txt', 'a+')
      for i in weights[0]:
          out_file.write(str(i)+'\n')
      out_file.close()
  
  elif choice == "Test":
  
      print("Serializing Test Data...")
  
      # SERIALIZE DATA
      file = open('train.txt')
      train_set = file.read().splitlines()
      positive = []
      negative = []
      for i in train_set:
          if (i[0] == '1' or i[0] == '2') and len(positive) < sample_size:
              positive.append(i)
          if (i[0] == '0') and len(negative) < sample_size:
              negative.append(i)
  
      test_set = positive+negative
  
      file = open('weights.txt', 'r').read().splitlines()
      weights = np.zeros((len(test_set), feature_num))
  
      # CREATE TEST SET
      for i in range(len(weights)):
          weights[i] = file
      features = []
      query = []
      output = np.zeros((len(test_set), feature_num))
      score = np.zeros((len(test_set)))
  
      for i in tqdm(range(len(test_set))):
          elements = test_set[i].split(' ')
          if int(elements[0]) == 0:
              score[i] = -1
          else:
              score[i] = 1
  
          query.append(int(elements[1].split(':')[1]))
          tmp = []
          for feature in elements[2:2+feature_num]:
              if feature.split(':')[1] == 'NULL':
                  tmp.append(0.0)
              else:
                  tmp.append(float(feature.split(':')[1]))
          features.append(tmp)
  
      features = np.asarray(features)
  
      for sample in range(len(test_set)):
          output[sample] = weights[sample]*features[sample]
          output[sample] = np.full((feature_num), np.sum(output[sample]))
  
      predictions = []
      for val in output:
          if(val[0] > 1):
              predictions.append(1)
          else:
              predictions.append(-1)
  
      print("-------------------------------------")
      print("Predicting...")
      print("-------------------------------------")
      print("Prediction finished with "+str(accuracy_score(score, predictions)*100)+"% accuracy.")
  
  
  My training algorithm
  if(val[0] >= 1):
      cost = 0
      weights = weights - alpha * (2 * 1/epochs * weights)
  else:
      cost = 1 - val[0]
      # WEIGHTS = WEIGHTS + LEARNING RATE * [X] * [Y]
      weights = weights + alpha * (features[count] * score[count] - 2 * 1/epochs * weights)
  
  
  What could I do to help the model train? Am I not giving it enough time? Is the algorithm wrong? Are the hyperparameters ok?
  Thanks for all your help.
  
  **Tags:** ai-design, classification, support-vector-machine
  > **Comments:**
  > **username_1**: Can you clarify this part "I classified 0 as -1 and 1, 2 as 1"? Does it mean that the original datasets has 3 classes and you're joining two classes so that you can perform binary classification? Also, you have only a total of 1k training examples? That's a relatively small dataset.  
  > **username_0**: Yes. The samples are scored 0 - 2, so I classified 0 as -1, and 1 and 2 as 1. Is this not a good way to do it? What would be an adequate sample size? Is my training algorithm ok?  
  
  ---
  
  ## 0 answers
  
  