Answers.Body: <p>It's called ill-conditioning in optimization, which means that you are a lot sensitive to changes with respect to some parameters than others.</p>
  <p><a href="https://i.sstatic.net/1rOqm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1rOqm.png" alt="enter image description here" /></a></p>
  <p>As you can see, the curvature is not the same in the X direction and in the Y direction (in your case, it would be w and m) and it's the reason why adaptive optimizers and momentum exists</p>
  <p>For linear regression, since the parameter space usually is small, you can also use the Limited-memory BFGS optimizer, or even the Newton Method, thus using 2-order informations to condition your step</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-09T09:22:45.643Z
Answers.Text: But isn't it obvious that the curvature will be different in x and y direction (m and b)? I never FOUND any real data where the curvature will be same in both direction! But in EVERY BOOK and LINKs I found they are using exact same learning rate for both. But why? For example, please see [link1](https://d2l.ai/chapter_linear-regression/linear-regression.html) and [link2](https://machinelearningmastery.com/training-a-linear-regression-model-in-pytorch/). Shouldn't it be more general to use learning parameters $\alpha_1$ and $\alpha_2$ instead of just $\alpha$?
Answers.UserDisplayName: Jahid Chowdhury Choton
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-09T11:11:23.957Z
Answers.Text: @JahidChowdhuryChoton what if you have 10 covariates? then you’ll have to manually tune 10 hyperparametrs… instead what is usually done is to set a stepsize that is fine for every parameter (which indeed is what optimization literature tells you to do with the Lipschitz constant)… in your case, it will (slowly but surely) converge to the minima if you use 0.000000001 for both of them
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-09T14:46:24.987Z
Answers.Text: Thank you for your valuable answer. I cannot upvote because my reputation is low. But I'll wait for a few more answers. You're right it will converge (but VERY SLOWLY) if I use 0.00000001 for both of them. But isn't it impractical since it is very slow? If I have 10 co-variates, it may become more and more impractical to use a single step size (which would be much more slower). Manually tuning can drastically reduce number of episodes. Can you suggest any reading material on optimization? Is it related to stability in systems?
Answers.UserDisplayName: Jahid Chowdhury Choton
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-09T19:08:57.717Z
Answers.Text: @JahidChowdhuryChoton it’s also impractical to tune manually each single one… instead we usually set an initial guess, and use information about the optimization history to change it (check SGD with momentum , Nesterov Momentum, and AdaGrad)
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-09T09:12:17.683Z
Answers.LastActivityDate: 2024-04-09T09:12:17.683Z
Answers.UserDisplayName: Alberto
Body: <p>I'm a new student in AI, currently learning linear regression. I used the <a href="https://www.kaggle.com/datasets/camnugent/california-housing-prices" rel="nofollow noreferrer">california housing dataset</a> for doing my experiments. My goal is to predict the 'population' column based on the 'total_rooms' column. I used the following formula and code to compute the slope 'm' and intercept 'b'.</p>
  <p><strong>Formula:</strong> <span class="math-container">$$ m = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2 }$$</span>
  <span class="math-container">$$ c = \bar{y} - m \bar{x} $$</span></p>
  <p>The code is as follows and it works perfectly:</p>
  <pre><code># Linear regression using the above formula
  x_vals = np.array(train['total_rooms'])
  y_vals = np.array(train['population'])
  xm = np.mean(x_vals)
  ym = np.mean(y_vals)
  def compute_m(x_vals, y_vals):    
      n = len(x_vals)
      sum_xy, sum_xx = 0, 0
      for i in range(n):
          sum_xy += (x_vals[i]-xm) * (y_vals[i]-ym)
          sum_xx += (x_vals[i]-xm)**2
      return sum_xy/sum_xx
  
  m = compute_m(x_vals, y_vals)
  c = ym - m*xm
  
  xl = np.array([np.min(x_vals), np.max(x_vals)])
  yl = m*xl + c
  plt.scatter(x_vals, y_vals)
  plt.plot(xl,yl, 'r')
  plt.show()
  print('m, c:', m,c)
  </code></pre>
  <p>To verify that my code is working. I also checked it with built-in linear regression in <strong>scikit-learn</strong> and it returns the exact same answer:</p>
  <pre><code># Now use scikit learn library
  from sklearn.linear_model import LinearRegression
  model = LinearRegression()
  ans = model.fit(x_vals.reshape(-1,1),y_vals.reshape(-1,1))
  score, intercept, coef = ans.score(x_vals.reshape(-1,1),y_vals.reshape(-1,1)), ans.intercept_, ans.coef_
  
  print('results from scikit-learn:', score, intercept, coef)
  </code></pre>
  <p>But the problem arises when I try to use gradient descent for learning the slope m and intercept c. My code is as follows:</p>
  <pre><code># Gradient descent
  def loss_func(y, ypred):
      mse = (y - ypred)**2
      return mse
  
  def gradient_loss(y, x, mc, bc):    
      n = len(y)
      print(n)
      m_loss, b_loss = 0, 0
      for i in range(n):
          ml = (-2/n) * x[i] * (y[i] - mc*x[i] - bc)
          m_loss += ml
          bl = (-2/n) * (y[i] - mc*x[i] - bc)
          b_loss += bl
      return m_loss, b_loss
  
  ep = 100000
  alpha = 0.000000001 # learning rate
  
  m, b = 0, 0
  for e in range(ep):
      y_pred = m*x_vals + b
      m_loss, b_loss = gradient_loss(y_vals, x_vals, m, b)
      print(&quot;m, b, m_loss, b_loss:&quot;, m, b, m_loss, b_loss)
      m = m - m_loss*alpha
      b = b - b_loss*0.001
  </code></pre>
  <p>I took the derivative of the squared loss, and using learning rate <span class="math-container">$\alpha$</span> for computing <span class="math-container">$m$</span> and intercept <span class="math-container">$b$</span> (sorry it is the same as <span class="math-container">$c$</span> in the previous block) for 100000 episodes. <strong>But note that I have to use different learning rate for m and b. No matter what learning rate I use, it does not work for both m and b. If I write in the last line 'b = b - b_loss*alpha', it never converges for any value of alpha</strong></p>
  <p>I've never seen in any article or book to use different learning rate for slope m and intercept b. Can anybody please explain what's happening here? And why I had to use different learning rate? :(</p>
  
Tags: <machine-learning><gradient-descent><linear-regression>
Title: Why same learning rate for slope and intercept not working in Linear regression?
UserDisplayName: Jahid Chowdhury Choton
ThreadText: # Why same learning rate for slope and intercept not working in Linear regression?
  
  (Asked by: username_0 on 2024-04-09)
  
  I'm a new student in AI, currently learning linear regression. I used the [california housing dataset](https://www.kaggle.com/datasets/camnugent/california-housing-prices) for doing my experiments. My goal is to predict the 'population' column based on the 'total\_rooms' column. I used the following formula and code to compute the slope 'm' and intercept 'b'.
  
  **Formula:** $$ m = \frac{\sum\_{i=1}^n (x\_i - \bar{x})(y\_i - \bar{y})}{\sum\_{i=1}^n (x\_i - \bar{x})^2 }$$
  $$ c = \bar{y} - m \bar{x} $$
  
  The code is as follows and it works perfectly:
  
  ```
  # Linear regression using the above formula
  x_vals = np.array(train['total_rooms'])
  y_vals = np.array(train['population'])
  xm = np.mean(x_vals)
  ym = np.mean(y_vals)
  def compute_m(x_vals, y_vals):    
      n = len(x_vals)
      sum_xy, sum_xx = 0, 0
      for i in range(n):
          sum_xy += (x_vals[i]-xm) * (y_vals[i]-ym)
          sum_xx += (x_vals[i]-xm)**2
      return sum_xy/sum_xx
  
  m = compute_m(x_vals, y_vals)
  c = ym - m*xm
  
  xl = np.array([np.min(x_vals), np.max(x_vals)])
  yl = m*xl + c
  plt.scatter(x_vals, y_vals)
  plt.plot(xl,yl, 'r')
  plt.show()
  print('m, c:', m,c)
  
  ```
  
  To verify that my code is working. I also checked it with built-in linear regression in **scikit-learn** and it returns the exact same answer:
  
  ```
  # Now use scikit learn library
  from sklearn.linear_model import LinearRegression
  model = LinearRegression()
  ans = model.fit(x_vals.reshape(-1,1),y_vals.reshape(-1,1))
  score, intercept, coef = ans.score(x_vals.reshape(-1,1),y_vals.reshape(-1,1)), ans.intercept_, ans.coef_
  
  print('results from scikit-learn:', score, intercept, coef)
  
  ```
  
  But the problem arises when I try to use gradient descent for learning the slope m and intercept c. My code is as follows:
  
  ```
  # Gradient descent
  def loss_func(y, ypred):
      mse = (y - ypred)**2
      return mse
  
  def gradient_loss(y, x, mc, bc):    
      n = len(y)
      print(n)
      m_loss, b_loss = 0, 0
      for i in range(n):
          ml = (-2/n) * x[i] * (y[i] - mc*x[i] - bc)
          m_loss += ml
          bl = (-2/n) * (y[i] - mc*x[i] - bc)
          b_loss += bl
      return m_loss, b_loss
  
  ep = 100000
  alpha = 0.000000001 # learning rate
  
  m, b = 0, 0
  for e in range(ep):
      y_pred = m*x_vals + b
      m_loss, b_loss = gradient_loss(y_vals, x_vals, m, b)
      print("m, b, m_loss, b_loss:", m, b, m_loss, b_loss)
      m = m - m_loss*alpha
      b = b - b_loss*0.001
  
  ```
  
  I took the derivative of the squared loss, and using learning rate $\alpha$ for computing $m$ and intercept $b$ (sorry it is the same as $c$ in the previous block) for 100000 episodes. **But note that I have to use different learning rate for m and b. No matter what learning rate I use, it does not work for both m and b. If I write in the last line 'b = b - b\_loss\*alpha', it never converges for any value of alpha**
  
  I've never seen in any article or book to use different learning rate for slope m and intercept b. Can anybody please explain what's happening here? And why I had to use different learning rate? :(
  
  **Tags:** machine-learning, gradient-descent, linear-regression
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-04-09. Score: 0 | Accepted answer
  
  It's called ill-conditioning in optimization, which means that you are a lot sensitive to changes with respect to some parameters than others.
  
  [![enter image description here](https://i.sstatic.net/1rOqm.png)](https://i.sstatic.net/1rOqm.png)
  
  As you can see, the curvature is not the same in the X direction and in the Y direction (in your case, it would be w and m) and it's the reason why adaptive optimizers and momentum exists
  
  For linear regression, since the parameter space usually is small, you can also use the Limited-memory BFGS optimizer, or even the Newton Method, thus using 2-order informations to condition your step
  
  > **Comments:**
  > **username_0**: But isn't it obvious that the curvature will be different in x and y direction (m and b)? I never FOUND any real data where the curvature will be same in both direction! But in EVERY BOOK and LINKs I found they are using exact same learning rate for both. But why? For example, please see [link1](https://d2l.ai/chapter_linear-regression/linear-regression.html) and [link2](https://machinelearningmastery.com/training-a-linear-regression-model-in-pytorch/). Shouldn't it be more general to use learning parameters $\alpha_1$ and $\alpha_2$ instead of just $\alpha$?  
  > **username_1**: @username_0 what if you have 10 covariates? then you’ll have to manually tune 10 hyperparametrs… instead what is usually done is to set a stepsize that is fine for every parameter (which indeed is what optimization literature tells you to do with the Lipschitz constant)… in your case, it will (slowly but surely) converge to the minima if you use 0.000000001 for both of them  
  > **username_0**: Thank you for your valuable answer. I cannot upvote because my reputation is low. But I'll wait for a few more answers. You're right it will converge (but VERY SLOWLY) if I use 0.00000001 for both of them. But isn't it impractical since it is very slow? If I have 10 co-variates, it may become more and more impractical to use a single step size (which would be much more slower). Manually tuning can drastically reduce number of episodes. Can you suggest any reading material on optimization? Is it related to stability in systems?  
  > **username_1**: @username_0 it’s also impractical to tune manually each single one… instead we usually set an initial guess, and use information about the optimization history to change it (check SGD with momentum , Nesterov Momentum, and AdaGrad)  
  
  ---
  
  