Answers.Body: <p>LogReg is not &quot;sensitive to magnitude&quot;, but the conclusions that you can take based on its learnt parameter are.</p>
  <p><span class="math-container">$$ 
  \begin{align}
  y = &amp; \beta^T x \\
  = &amp; (\beta / 1000)^T (x \cdot 1000) \\
  \end{align}
  $$</span></p>
  <p>However, if you base your &quot;feature selection&quot; on <span class="math-container">$\beta$</span>, with the reasoning &quot;bogger weights = more importance&quot;, then this invariance is not there anymore, and the magnitude of the feature matters a lot</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-09T16:42:49.500Z
Answers.LastActivityDate: 2024-09-09T16:42:49.500Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>It is true that if a feature has a large magnitude, the corresponding weight learned by the logistic regression model will tend to be smaller to compensate and balance the overall output. However, this balancing act doesn't always work well in practice, and it can lead to issues such as ill-conditioned optimization.</p>
  <p>In logistic regression, the weight represents the change in the log-odds of the target class given a one-unit change in the feature. If the scale of a feature is large, its weight learned will need to be extremely small to compensate as you correctly claimed which makes the gradient descent steps inefficient and convergence to the optimal weights becomes slower or unstable.</p>
  <p>Therefore usually normalizing or standardizing features is crucial to mitigate these problems, ensuring that the optimization process is well-behaved and the model is more interpretable and robust.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-10T05:16:38.983Z
Answers.LastActivityDate: 2024-09-10T05:16:38.983Z
Answers.UserDisplayName: cinch
Body: <p>I have tried to understand from various sources but i was unable to grasp how logistic regression is sensitive to magnitude of feature.</p>
  <p>If the feature is large then weights will be smaller thus balancing the overall output. Can anyone explain this in depth?</p>
  
Tags: <machine-learning><logistic-regression>
Title: how logistic regression is sensitive to magnitude of feature?
UserDisplayName: Gargi Bindal
ThreadText: # how logistic regression is sensitive to magnitude of feature?
  
  (Asked by: username_0 on 2024-09-09)
  
  I have tried to understand from various sources but i was unable to grasp how logistic regression is sensitive to magnitude of feature.
  
  If the feature is large then weights will be smaller thus balancing the overall output. Can anyone explain this in depth?
  
  **Tags:** machine-learning, logistic-regression
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-09-09. Score: 0
  
  LogReg is not "sensitive to magnitude", but the conclusions that you can take based on its learnt parameter are.
  
  $$
  \begin{align}
  y = & \beta^T x \\
  = & (\beta / 1000)^T (x \cdot 1000) \\
  \end{align}
  $$
  
  However, if you base your "feature selection" on $\beta$, with the reasoning "bogger weights = more importance", then this invariance is not there anymore, and the magnitude of the feature matters a lot
  
  
  ---
  
  ### Answer by username_2 on 2024-09-10. Score: 0
  
  It is true that if a feature has a large magnitude, the corresponding weight learned by the logistic regression model will tend to be smaller to compensate and balance the overall output. However, this balancing act doesn't always work well in practice, and it can lead to issues such as ill-conditioned optimization.
  
  In logistic regression, the weight represents the change in the log-odds of the target class given a one-unit change in the feature. If the scale of a feature is large, its weight learned will need to be extremely small to compensate as you correctly claimed which makes the gradient descent steps inefficient and convergence to the optimal weights becomes slower or unstable.
  
  Therefore usually normalizing or standardizing features is crucial to mitigate these problems, ensuring that the optimization process is well-behaved and the model is more interpretable and robust.
  
  
  ---
  
  