Answers.Body: <p>The higher (or smaller) the learning rate, the higher (or, respectively, smaller) the contribution of the gradient of the objective function, with respect to the parameters of the model, to the new parameters of the model. Therefore, if you progressively increase (or decrease) the learning rate, then you will accelerate (or, respectively, slow down) the learning process, so later training examples have higher (or, respectively, smaller) influence on the parameters of the model.</p>
  
  <p>In your example, the function <code>warmup_lr_scheduler</code> returns an object of class <code>LambdaLR</code>, initialized with a certain optimizer and the function <code>f</code>, which is defined as</p>
  
  <pre><code>def f(x):
      if x &gt;= warmup_iters:
          return 1
      alpha = float(x) / warmup_iters
      return warmup_factor * (1 - alpha) + alpha
  </code></pre>
  
  <p>The documentation of <code>torch.optim.lr_scheduler.LambdaLR</code> says that the function <code>f</code> should compute a multiplicative factor given an integer parameter epoch, so <code>x</code> is a training epoch. If the epoch <code>x</code> is greater than or equal to <code>warmup_iters</code>, then 1 is returned, but anything multiplied by 1 is itself, so, when the epoch <code>x</code> is greater than a threshold, <code>warmup_iters</code> (e.g. 1000), then the initial learning rate is unaffected. However, when <code>x &lt; warmup_iters</code>, the multiplicative factor is given by</p>
  
  <pre><code>alpha = float(x) / warmup_iters
  warmup_factor * (1 - alpha) + alpha
  </code></pre>
  
  <p>which is a function of the epoch <code>x</code>. The higher the epoch <code>x</code>, the higher the value of <code>alpha</code>, so the smaller <code>(1 - alpha)</code> and <code>warmup_factor * (1 - alpha)</code>. Note that <code>float(x) / warmup_iters</code> will never be greater than 1 because <code>x</code> is never greater than <code>warmup_iters</code>. So, effectively, as the epoch increases, <code>warmup_factor * (1 - alpha)</code> tends to 0 and <code>alpha</code> tends to 1. </p>
  
  <p>The learning rate can only increase if you multiply it with a constant greater than 1. However, this can only happen if <code>warmup_factor &gt; 1</code>. You can verify this by solving the inequality <code>warmup_factor * (1 - alpha) + alpha &gt; 1</code>.</p>
  
  <p>To conclude, the initial learning rate is not being increased, but the learning process starts with a smaller learning rate than the given learning rate, for a <code>warmup_iters</code> epochs, then, after <code>warmup_iters</code> epochs, it uses the initially given learning rate (e.g. 0.002).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-12T15:50:27.620Z
Answers.LastActivityDate: 2019-10-12T15:56:17.190Z
Answers.LastEditDate: 2019-10-12T15:56:17.190Z
Answers.UserDisplayName: nbro
Body: <p>I was reading a PyTorch code then I saw this learning rate scheduler:</p>
  
  <pre><code>def warmup_lr_scheduler(optimizer, warmup_iters, warmup_factor):
      """
      Learning rate scheduler
      :param optimizer:
      :param warmup_iters:
      :param warmup_factor:
      :return:
      """
      def f(x):
          if x &gt;= warmup_iters:
              return 1
          alpha = float(x) / warmup_iters
          return warmup_factor * (1 - alpha) + alpha
  
      return torch.optim.lr_scheduler.LambdaLR(optimizer, f)
  </code></pre>
  
  <p>and this is where the function is called:</p>
  
  <pre><code>if epoch == 0:
      warmup_factor = 1. / 1000
      warmup_iters = min(1000, len(data_loader) - 1)
  
      lr_scheduler = utils.warmup_lr_scheduler(optimizer, warmup_iters, warmup_factor)
  </code></pre>
  
  <p>As I understood it gradually increase learning rate until it reach initial learning rate. Am I correct? Why we need to increase learning rate? As I know for better learning in Neural Networks we decrease learning rate. </p>
  
Tags: <neural-networks><machine-learning><deep-learning><pytorch><learning-rate>
Title: Is this learning rate schedule increasing the learning rate?
UserDisplayName: CVDE
ThreadText: # Is this learning rate schedule increasing the learning rate?
  
  (Asked by: username_0 on 2019-10-12)
  
  I was reading a PyTorch code then I saw this learning rate scheduler:
  def warmup_lr_scheduler(optimizer, warmup_iters, warmup_factor):
      """
      Learning rate scheduler
      :param optimizer:
      :param warmup_iters:
      :param warmup_factor:
      :return:
      """
      def f(x):
          if x >= warmup_iters:
              return 1
          alpha = float(x) / warmup_iters
          return warmup_factor * (1 - alpha) + alpha
  
      return torch.optim.lr_scheduler.LambdaLR(optimizer, f)
  
  and this is where the function is called:
  if epoch == 0:
      warmup_factor = 1. / 1000
      warmup_iters = min(1000, len(data_loader) - 1)
  
      lr_scheduler = utils.warmup_lr_scheduler(optimizer, warmup_iters, warmup_factor)
  
  As I understood it gradually increase learning rate until it reach initial learning rate. Am I correct? Why we need to increase learning rate? As I know for better learning in Neural Networks we decrease learning rate.
  
  **Tags:** neural-networks, machine-learning, deep-learning, pytorch, learning-rate
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-10-12. Score: 1 | Accepted answer
  
  The higher (or smaller) the learning rate, the higher (or, respectively, smaller) the contribution of the gradient of the objective function, with respect to the parameters of the model, to the new parameters of the model. Therefore, if you progressively increase (or decrease) the learning rate, then you will accelerate (or, respectively, slow down) the learning process, so later training examples have higher (or, respectively, smaller) influence on the parameters of the model.
  In your example, the function warmup_lr_scheduler returns an object of class LambdaLR, initialized with a certain optimizer and the function f, which is defined as
  def f(x):
      if x >= warmup_iters:
          return 1
      alpha = float(x) / warmup_iters
      return warmup_factor * (1 - alpha) + alpha
  
  The documentation of torch.optim.lr_scheduler.LambdaLR says that the function f should compute a multiplicative factor given an integer parameter epoch, so x is a training epoch. If the epoch x is greater than or equal to warmup_iters, then 1 is returned, but anything multiplied by 1 is itself, so, when the epoch x is greater than a threshold, warmup_iters (e.g. 1000), then the initial learning rate is unaffected. However, when x < warmup_iters, the multiplicative factor is given by
  alpha = float(x) / warmup_iters
  warmup_factor * (1 - alpha) + alpha
  
  which is a function of the epoch x. The higher the epoch x, the higher the value of alpha, so the smaller (1 - alpha) and warmup_factor * (1 - alpha). Note that float(x) / warmup_iters will never be greater than 1 because x is never greater than warmup_iters. So, effectively, as the epoch increases, warmup_factor * (1 - alpha) tends to 0 and alpha tends to 1. 
  The learning rate can only increase if you multiply it with a constant greater than 1. However, this can only happen if warmup_factor > 1. You can verify this by solving the inequality warmup_factor * (1 - alpha) + alpha > 1.
  To conclude, the initial learning rate is not being increased, but the learning process starts with a smaller learning rate than the given learning rate, for a warmup_iters epochs, then, after warmup_iters epochs, it uses the initially given learning rate (e.g. 0.002).
  
  
  ---
  
  