Body: <p>I am trying to solve a classification problem by implementing the Least Squares algorithm in Python. To solve this problem, I am implementing the linear algebra formula to train the classifier, which is <span class="math-container">$w = (X^TX)X^Ty$</span>, where <span class="math-container">$w$</span> is the final weight vector of the classification function, <span class="math-container">$X$</span> is an input matrix of training data and <span class="math-container">$y$</span> a matrix of training labels. The classifier must be able to classify into three classes. As seen in the following snippet, during the preparation of the data, I gather my training data in matrix <span class="math-container">$X$</span>, adding the number one at the end of each sample. I also gather my training labels in matrix <span class="math-container">$y$</span>, coding each class as a sequence of -1 and 1.</p>
  <pre><code>    X = np.matrix(np.zeros((len(train_set),4)))
      y = np.matrix(np.zeros((len(train_set),3)))
  
      for i, row in train_set.iterrows():
          X[i] = [row[1], row[2], row[3], 1]
          if row[0] == 'H':
              y[i] = [1, -1, -1]
          elif row[0] == 'D':
              y[i] = [-1, 1, -1]
          else:
              y[i] = [-1, -1, 1]
  </code></pre>
  <p>What we have in matrix <span class="math-container">$y$</span> in the end, is a matrix that each column is a representation of each class and can tell us which samples belong to the class of the corresponding column. Having explained what each matrix in my program contains, my question is this, which of the following two implementations of the training process is correct and why?</p>
  <p>At first, I went with the implementation seen in the snippet below.</p>
  <pre><code>    Xtranspose = X.T
      dotProduct = Xtranspose.dot(X)
      inverse = np.linalg.pinv(dotProduct)
      A = inverse.dot(Xtranspose)
      w = A.dot(y)
      
      for i, row in test_set.iterrows():
          r = np.matrix([row[1], row[2], row[3], 1]).dot(w)
  </code></pre>
  <p>As you can see, considering that <span class="math-container">$A = (X^TX)X^T$</span>, I multiply <span class="math-container">$A$</span> with the label matrix <span class="math-container">$y$</span> and then I use the weight vector <span class="math-container">$w$</span> in the loop to test the classifier on some test data. Later, though, after some research on the internet, I found this second implementation, which actually has a higher success rate.</p>
  <pre><code>    for i, row in test_set.iterrows():
          r = np.zeros([3])
          j = 0
          for column in y.T:
              w = A.dot(column.T)
              r[j] = np.matrix([row[1], row[2], row[3], 1]).dot(w)
              j += 1
  </code></pre>
  <p>Having calculated <span class="math-container">$A$</span> beforehand, I now calculate the weights for each column of <span class="math-container">$y$</span>, for each class, separately. This second method, has a 10% greater success rate than the first one. So, why does the second training method have a better success rate? Is the second method the right training method?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-21T21:47:48.077Z
Comments.Text: why least squares for regression?
Comments.UserDisplayName: David
Tags: <classification><python><implementation><linear-regression>
Title: Which of the following two implementations of a Least Squares classifier in Python is correct?
UserDisplayName: User9123
ThreadText: # Which of the following two implementations of a Least Squares classifier in Python is correct?
  
  (Asked by: username_0 on 2021-08-31)
  
  I am trying to solve a classification problem by implementing the Least Squares algorithm in Python. To solve this problem, I am implementing the linear algebra formula to train the classifier, which is $w = (X^TX)X^Ty$, where $w$ is the final weight vector of the classification function, $X$ is an input matrix of training data and $y$ a matrix of training labels. The classifier must be able to classify into three classes. As seen in the following snippet, during the preparation of the data, I gather my training data in matrix $X$, adding the number one at the end of each sample. I also gather my training labels in matrix $y$, coding each class as a sequence of -1 and 1.
      X = np.matrix(np.zeros((len(train_set),4)))
      y = np.matrix(np.zeros((len(train_set),3)))
  
      for i, row in train_set.iterrows():
          X[i] = [row[1], row[2], row[3], 1]
          if row[0] == 'H':
              y[i] = [1, -1, -1]
          elif row[0] == 'D':
              y[i] = [-1, 1, -1]
          else:
              y[i] = [-1, -1, 1]
  
  What we have in matrix $y$ in the end, is a matrix that each column is a representation of each class and can tell us which samples belong to the class of the corresponding column. Having explained what each matrix in my program contains, my question is this, which of the following two implementations of the training process is correct and why?
  At first, I went with the implementation seen in the snippet below.
      Xtranspose = X.T
      dotProduct = Xtranspose.dot(X)
      inverse = np.linalg.pinv(dotProduct)
      A = inverse.dot(Xtranspose)
      w = A.dot(y)
      
      for i, row in test_set.iterrows():
          r = np.matrix([row[1], row[2], row[3], 1]).dot(w)
  
  As you can see, considering that $A = (X^TX)X^T$, I multiply $A$ with the label matrix $y$ and then I use the weight vector $w$ in the loop to test the classifier on some test data. Later, though, after some research on the internet, I found this second implementation, which actually has a higher success rate.
      for i, row in test_set.iterrows():
          r = np.zeros([3])
          j = 0
          for column in y.T:
              w = A.dot(column.T)
              r[j] = np.matrix([row[1], row[2], row[3], 1]).dot(w)
              j += 1
  
  Having calculated $A$ beforehand, I now calculate the weights for each column of $y$, for each class, separately. This second method, has a 10% greater success rate than the first one. So, why does the second training method have a better success rate? Is the second method the right training method?
  
  **Tags:** classification, python, implementation, linear-regression
  > **Comments:**
  > **username_1**: why least squares for regression?  
  
  ---
  
  ## 0 answers
  
  