title: What is wrong in this Python code for Regularized Linear Regression?
body: <p>I wrote code with numpy(theta, X is numpy array):</p>
  
  <pre><code>def CostRegFunction(X, y, theta, lambda_):
      m = len(X)
      # add bias unit
      X = np.concatenate((np.ones((m,1)),X),1)
  
      H = np.dot(X,theta)
  
      J = (1 / (2 * m)) * (np.sum([(H[i] - y[i][0])**2 for i in range(len(H))])) + (lambda_ / (2 * m)) * np.sum(theta[1:]**2)
  
      grad_ = list()
  
      grad_.append((1 / m) * np.sum([(H[j] - y[j][0]) for j in range(len(H))]))
      for i in range(len(theta)-1):
          grad_.append((1 / m) * np.sum([(H[j] - y[j]) * X[j][i+1] for j in range(len(H))]) + (lambda_ / m) * theta[i+1])
  
      return J, grad_
  
  
  
  def TrainLinearReg(X, y, theta, lambda_, alpha, iter):
  
      JHistory = list()
      for i in range(iter):
          J, grad = CostRegFunction(X, y, theta, Lambda_)
          JHistory.append(J)
          for j in range(len(theta)):
              theta[j] = theta[j] - alpha * grad[j]
  
      return theta, JHistory
  
  Theta, JH = TrainLinearReg(X, y, th, Lambda_, 0.01, 50)
  </code></pre>
  
  <p>But when I try learn theta this code gives me a realy huge grow of theta and value of J.
  For example first iteration grad = [-15.12452, 598.435436] - it is correct. J is 303.3255
  2nd iteration - grad = [10.23566,-3646.2345] J = 7924
  and so on J grows faster and faster but on idea of LR it must be lower.</p>
  
  <p>But if I use Normal Linear Equation in gives me a good Theta.</p>
  
  <p>What is wrong in that code?</p>