Answers.Body: <p>Gradient Descent is a method to find the optimum parameter of the hypothesis or minimize the cost function.</p>
  
  <p><a href="https://i.sstatic.net/UW6mo.png" rel="noreferrer"><img src="https://i.sstatic.net/UW6mo.png" alt="formula"></a>
  where alpha is learning rate </p>
  
  <p>If the learning rate is high then it can overshoot the minimum and can fail to minimize the cost function.
  <a href="https://i.sstatic.net/w7Y3c.png" rel="noreferrer"><img src="https://i.sstatic.net/w7Y3c.png" alt="enter image description here"></a></p>
  
  <p>hence result in a higher loss.</p>
  
  <p><a href="https://i.sstatic.net/qwqsM.png" rel="noreferrer"><img src="https://i.sstatic.net/qwqsM.png" alt="enter image description here"></a></p>
  
  <p>Since Gradient descent can only find local minimum so, the lower learning rate may result in bad performance. To do so, it is better to start with the random value of the hyperparameter can increase model training time but there are advanced methods such as adaptive gradient descent can manage the training time.</p>
  
  <p>There are lots of optimizer for the same task but no optimizer is perfect. It depends on some factors</p>
  
  <ol>
  <li>size of training data: as the size of the training data increases training time for model increases. If you want to go with less training model time you can choose a higher learning rate but may result in bad performance.</li>
  <li>Optimizer(gradient descent) will be slow down whenever the gradient is small then it is better to go with a higher learning rate.</li>
  </ol>
  
  <p>PS. It is always better to go with different rounds of gradient descent</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T13:29:43.773Z
Answers.Text: This is good start, as it shows the difference between low and high learning rates in general. You also need to explain why the good learning rate varies depending on the task - and the OP was specifically asking why some problems require a lower learning rate than others
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T14:13:15.497Z
Answers.Text: That's a good point. I have edited it. Since there is not a specific problem is mention I am going with general one.
Answers.UserDisplayName: Posi2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T14:36:13.037Z
Answers.Text: I still think that this does not answer the question. The OP is not asking about the optimiser or data, it is asking about the model. How does the model (its architecture, number of parameters, etc.) affect the learning rate? I think this is the actual question, which you do not answer. Everything else is quite irrelevant to the question and will only confuse readers that can't distinguish between these concepts.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T16:44:17.393Z
Answers.Text: Thanks for the feedback. Irrespective of the model architecture as the number of the parameter, size of data and range of the data (solution use normalized data) is high result in the higher training time so according to it, we should change the learning rate. This applies for the model such as linear regression, logistic regression, SVM etc since they use GD for optimization. Any response is always welcome :)
Answers.UserDisplayName: Posi2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T17:23:24.183Z
Answers.Text: Any proof that assesses your claim "irrespective of the model architecture"? This answer still does not answer the OP question. You're answering to the question "how does the learning rate change in general, depending on the machine learning setting" (and your answer is not exhaustive, of course, because it does not mention "how the learning rate changes depending on the model", i.e. the actual question).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T18:28:56.830Z
Answers.Text: What is machine learning setting? Gradient descent is algorithm which takes some parameter, Algorithm does not change with change in model it remains same. Of course there is change in number of parameter, size, range of value etc.
Answers.UserDisplayName: Posi2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-02T04:02:22.583Z
Answers.Text: I mentioned that gradient descent find out local minimum by considering non convex curve and there are also other better optimizer are present.
Answers.UserDisplayName: Posi2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T13:27:29.457Z
Answers.LastActivityDate: 2019-04-01T14:11:39.637Z
Answers.LastEditDate: 2019-04-01T14:11:39.637Z
Answers.UserDisplayName: Posi2
Body: <p>I've pondered this for a while without developing an intuition for the math behind the cause of this.  </p>
  
  <p>So what causes a model to need a low learning rate?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-31T22:11:47.167Z
Comments.Text: I wondered about it too and I am curious why RNNs have a smaller learning rate than CNNs. From what I know, model complexity (deepness) and/or huge datasets require a finer hyperparameter for the lr.
Comments.UserDisplayName: iustin
Tags: <machine-learning><models><hyper-parameters><learning-rate>
Title: What causes a model to require a low learning rate?
UserDisplayName: JohnAllen
ThreadText: # What causes a model to require a low learning rate?
  
  (Asked by: username_0 on 2019-03-31)
  
  I've pondered this for a while without developing an intuition for the math behind the cause of this.
  
  So what causes a model to need a low learning rate?
  
  **Tags:** machine-learning, models, hyper-parameters, learning-rate
  > **Comments:**
  > **username_1**: I wondered about it too and I am curious why RNNs have a smaller learning rate than CNNs. From what I know, model complexity (deepness) and/or huge datasets require a finer hyperparameter for the lr.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-04-01. Score: 6
  
  Gradient Descent is a method to find the optimum parameter of the hypothesis or minimize the cost function.
  
  [![formula](https://i.sstatic.net/UW6mo.png)](https://i.sstatic.net/UW6mo.png)
  where alpha is learning rate
  
  If the learning rate is high then it can overshoot the minimum and can fail to minimize the cost function.
  [![enter image description here](https://i.sstatic.net/w7Y3c.png)](https://i.sstatic.net/w7Y3c.png)
  
  hence result in a higher loss.
  
  [![enter image description here](https://i.sstatic.net/qwqsM.png)](https://i.sstatic.net/qwqsM.png)
  
  Since Gradient descent can only find local minimum so, the lower learning rate may result in bad performance. To do so, it is better to start with the random value of the hyperparameter can increase model training time but there are advanced methods such as adaptive gradient descent can manage the training time.
  
  There are lots of optimizer for the same task but no optimizer is perfect. It depends on some factors
  
  1. size of training data: as the size of the training data increases training time for model increases. If you want to go with less training model time you can choose a higher learning rate but may result in bad performance.
  2. Optimizer(gradient descent) will be slow down whenever the gradient is small then it is better to go with a higher learning rate.
  
  PS. It is always better to go with different rounds of gradient descent
  
  > **Comments:**
  > **username_3**: This is good start, as it shows the difference between low and high learning rates in general. You also need to explain why the good learning rate varies depending on the task - and the OP was specifically asking why some problems require a lower learning rate than others  
  > **username_2**: That's a good point. I have edited it. Since there is not a specific problem is mention I am going with general one.  
  > **username_4**: I still think that this does not answer the question. The OP is not asking about the optimiser or data, it is asking about the model. How does the model (its architecture, number of parameters, etc.) affect the learning rate? I think this is the actual question, which you do not answer. Everything else is quite irrelevant to the question and will only confuse readers that can't distinguish between these concepts.  
  > **username_2**: Thanks for the feedback. Irrespective of the model architecture as the number of the parameter, size of data and range of the data (solution use normalized data) is high result in the higher training time so according to it, we should change the learning rate. This applies for the model such as linear regression, logistic regression, SVM etc since they use GD for optimization. Any response is always welcome :)  
  > **username_4**: Any proof that assesses your claim "irrespective of the model architecture"? This answer still does not answer the OP question. You're answering to the question "how does the learning rate change in general, depending on the machine learning setting" (and your answer is not exhaustive, of course, because it does not mention "how the learning rate changes depending on the model", i.e. the actual question).  
  > **username_2**: What is machine learning setting? Gradient descent is algorithm which takes some parameter, Algorithm does not change with change in model it remains same. Of course there is change in number of parameter, size, range of value etc.  
  > **username_2**: I mentioned that gradient descent find out local minimum by considering non convex curve and there are also other better optimizer are present.  
  
  ---
  
  