Answers.Body: <p>What are you hoping to get out of the answer for this question?
  Feature scaling is a method you CAN(but don't have to) use, so that your algorithm performs faster and reaches better general accuracy.
  I would say that on a simple regression task, where the feature value ranges do not vary a lot, the output would probably be almost the same, but as soon as you introduce data that has one feature of range 1-10, and other range of 10000-100000, that is where you would notice that you NEED to standardize/normalize your features in order to reach optimal results.
  That's why it's almost a general rule to just scale your data, so your algorithm can generalize better, and you don't have to worry about your algorithm giving higher importance to a feature with higher values, instead of the one with lower values(just an example).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-31T10:32:05.490Z
Answers.Text: I am hoping to know whether the gradient descent is scale invariant or not
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-31T10:29:05.647Z
Answers.LastActivityDate: 2020-07-31T10:29:05.647Z
Answers.UserDisplayName: Ivan Petrovic Markovic
Body: <p>I know we should scale the input and output (assuming regression task) before we feed it to the neural network. Then the gradient descent will give the better minima much faster. But I have subtle confusion whether gradient descent with feature scale and without feature scale gives the same result or just gradient descent is not scale-invariant.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-31T12:37:11.213Z
Comments.Text: More precisely, what do you mean by "scale-invariant" in this case? Gradient descent/ascent is a general method to optimize a function that is differentiable. We usually talk about functions and their gradients. In the context of neural networks, the functions are the loss/error functions.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-07T22:19:36.107Z
Comments.Text: "scale-invariant" in the sense, if I rescale my loss function ( normalization or standardization) it'll give me the same result.
Comments.UserDisplayName: Swakshar Deb
Tags: <machine-learning><deep-learning>
Title: Is gradient descent scale invariant or not?
UserDisplayName: Swakshar Deb
ThreadText: # Is gradient descent scale invariant or not?
  
  (Asked by: username_0 on 2020-07-31)
  
  I know we should scale the input and output (assuming regression task) before we feed it to the neural network. Then the gradient descent will give the better minima much faster. But I have subtle confusion whether gradient descent with feature scale and without feature scale gives the same result or just gradient descent is not scale-invariant.
  
  **Tags:** machine-learning, deep-learning
  
  > **Comments:**
  > **username_1**: More precisely, what do you mean by "scale-invariant" in this case? Gradient descent/ascent is a general method to optimize a function that is differentiable. We usually talk about functions and their gradients. In the context of neural networks, the functions are the loss/error functions.  
  > **username_0**: "scale-invariant" in the sense, if I rescale my loss function ( normalization or standardization) it'll give me the same result.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-31. Score: -1
  
  What are you hoping to get out of the answer for this question?
  Feature scaling is a method you CAN(but don't have to) use, so that your algorithm performs faster and reaches better general accuracy.
  I would say that on a simple regression task, where the feature value ranges do not vary a lot, the output would probably be almost the same, but as soon as you introduce data that has one feature of range 1-10, and other range of 10000-100000, that is where you would notice that you NEED to standardize/normalize your features in order to reach optimal results.
  That's why it's almost a general rule to just scale your data, so your algorithm can generalize better, and you don't have to worry about your algorithm giving higher importance to a feature with higher values, instead of the one with lower values(just an example).
  
  > **Comments:**
  > **username_0**: I am hoping to know whether the gradient descent is scale invariant or not  
  
  ---
  
  