Answers.Body: <p>If the learning rate is greater than or equal to <span class="math-container">$1$</span> the Robbins-Monro condition
  <span class="math-container">$$\sum _{{t=0}}^{{\infty }}a_{t}^{2}&lt;\infty\label{1}\tag{1},$$</span></p>
  <p>where <span class="math-container">$a_t$</span> is the learning rate at iteration <span class="math-container">$t$</span>, does not hold (given that a number bigger than <span class="math-container">$1$</span> squared becomes a bigger number), so stochastic gradient descent is not generally guaranteed to converge to a minimum [<a href="https://arxiv.org/pdf/1401.0118.pdf" rel="noreferrer">1</a>] (although the condition \ref{1} is a sum from <span class="math-container">$t=0$</span> to <span class="math-container">$t=\infty$</span>, but, of course, we only iterate for a finite number of iterations). Moreover, note that, if the learning rate is bigger than <span class="math-container">$1$</span>, you are essentially giving more weight to the gradient of the loss function than to the current value of the parameters (you give weight <span class="math-container">$1$</span> to the parameters).</p>
  <p>This is probably the main reason why the learning rate is usually in the range <span class="math-container">$(0, 1)$</span> and there are methods to decay the learning rate, which can be beneficial (and there are several explanations of why this is the case [<a href="https://arxiv.org/pdf/1908.01878.pdf" rel="noreferrer">2</a>]).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-25T13:22:54.027Z
Answers.LastActivityDate: 2020-09-25T13:28:25.827Z
Answers.LastEditDate: 2020-09-25T13:28:25.827Z
Answers.UserDisplayName: nbro
Body: <p>In all examples I've ever seen, the learning rate of an optimisation method is always less than <span class="math-container">$1$</span>. However, I've never found an explanation as to why this is. In addition to that, there are some cases where having a learning rate bigger than 1 is beneficial, such as in the case of <a href="https://arxiv.org/abs/1708.07120" rel="noreferrer">super-convergence</a>.</p>
  <p>Why is the learning rate generally less than 1? Specifically, when performing an update on a parameter, why is the gradient generally multiplied by a factor less than 1 (absolutely)?</p>
  
Tags: <machine-learning><optimization><gradient-descent><learning-rate><stochastic-gradient-descent>
Title: Why is the learning rate generally beneath 1?
UserDisplayName: Recessive
ThreadText: # Why is the learning rate generally beneath 1?
  
  (Asked by: username_0 on 2020-09-25)
  
  In all examples I've ever seen, the learning rate of an optimisation method is always less than $1$. However, I've never found an explanation as to why this is. In addition to that, there are some cases where having a learning rate bigger than 1 is beneficial, such as in the case of [super-convergence](https://arxiv.org/abs/1708.07120).
  
  Why is the learning rate generally less than 1? Specifically, when performing an update on a parameter, why is the gradient generally multiplied by a factor less than 1 (absolutely)?
  
  **Tags:** machine-learning, optimization, gradient-descent, learning-rate, stochastic-gradient-descent
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-25. Score: 11 | Accepted answer
  
  If the learning rate is greater than or equal to $1$ the Robbins-Monro condition
  $$\sum \_{{t=0}}^{{\infty }}a\_{t}^{2}<\infty\label{1}\tag{1},$$
  
  where $a\_t$ is the learning rate at iteration $t$, does not hold (given that a number bigger than $1$ squared becomes a bigger number), so stochastic gradient descent is not generally guaranteed to converge to a minimum [[1](https://arxiv.org/pdf/1401.0118.pdf)] (although the condition \ref{1} is a sum from $t=0$ to $t=\infty$, but, of course, we only iterate for a finite number of iterations). Moreover, note that, if the learning rate is bigger than $1$, you are essentially giving more weight to the gradient of the loss function than to the current value of the parameters (you give weight $1$ to the parameters).
  
  This is probably the main reason why the learning rate is usually in the range $(0, 1)$ and there are methods to decay the learning rate, which can be beneficial (and there are several explanations of why this is the case [[2](https://arxiv.org/pdf/1908.01878.pdf)]).
  
  
  ---
  
  