Answers.Body: <p>I'll start by saying that optimization is as much art as it is science.</p>
  <p><a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html" rel="nofollow noreferrer">ReduceLROnPlateau</a> is my favourite scheduler and I've found it performs quite well out of the box. Of course it's good to play with it to achieve optimal results but for what you experience there might be other reasons.</p>
  <p>Depending on the size of your data, 100 epochs might be more than enough so this might not be a problem to begin with.</p>
  <p>In case this is indeed a problem, since you don't give much details of your training, I can only provide some generic suggestions.</p>
  <p>Check if your model learns without a scheduler. Find some good hyperparameters for:</p>
  <ul>
  <li><p>The starting learning rate. It might be too small to begin with, or even too big (this could mess the network initialization).</p>
  </li>
  <li><p>The batch size cause it could greatly affect training dynamics.</p>
  </li>
  <li><p>Any other parameter you think it's important.</p>
  </li>
  </ul>
  <p>After you have these, then add the scheduler. If you still have issues try one of the following:</p>
  <ul>
  <li><p>Check if the metric that you're monitoring is the right one, i.e. a loss on a validation set and that it's values make sense.</p>
  </li>
  <li><p>Check the iterations per epoch. A very small number of iterations per epoch could give noisy validation metrics.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-13T22:22:41.873Z
Answers.Text: I also like the idea of it, but this is the 2nd time I've found it's defaults problematic because in practice I usually want to train my models for thousands of epochs.
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-14T17:21:51.920Z
Answers.Text: It's hard to help more without knowning the specifics. Consider sharing a bit more information about your model, data and training.
Answers.UserDisplayName: sfotiadis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-20T18:42:42.297Z
Answers.Text: Thanks I think I discovered the problem to be that epochs aren't a proper "unit" because the number of gradient steps that they represent can vary by orders of magnitude depending on the size of the dataset. And these defaults are probably designed for enormous image datasets with many gradient steps.
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-22T18:43:47.387Z
Answers.Text: Good to know you solved it. Yeah these methods target the stochastic gradient descent regime. If you're doing only a small amount of iterations per epoch then measuring the validation error in every epoch can be too frequent, your model doesn't have time to become better (hence noisy validation metrics). It's probably not very efficient either.
Answers.UserDisplayName: sfotiadis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-22T18:48:39.403Z
Answers.Text: Ya your comment about 100 epochs "being more than enough", was enough for me to figure out the rest. Thanks
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-13T21:02:33.603Z
Answers.LastActivityDate: 2024-03-13T21:02:33.603Z
Answers.UserDisplayName: sfotiadis
Body: <p>Hi I'm asking about the learning rate schedule <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html" rel="nofollow noreferrer">ReduceLROnPlateau()</a>. I've noticed that it decreases the learning rate by a factor of 10 each time and by default the patience is 10. Given phenomena like <a href="https://en.wikipedia.org/wiki/Double_descent" rel="nofollow noreferrer">double descent</a> its default parameters are very concerning.</p>
  <p>Not to mention that in practice I find it to be very common that learning might stall for 10 epochs (even if not truly stuck). And for example this LR schedule <strong>(with patience=6)</strong> gives me <strong>learning_rate=2e-9 after just 100 epochs!!</strong> Presumably it would further decrease it if not for the epsilon parameter, which isn't intuitive as you would think it should decrease gradually...</p>
  <p>Are they basing these defaults off of some paper? Can anyone speak to their experience using this schedule &amp; the performance of its default settings? And/or share relevant papers.</p>
  
Tags: <learning-rate>
Title: Is ReduceLROnPlateau() Schedule too aggressive by default in pytorch?
UserDisplayName: profPlum
ThreadText: # Is ReduceLROnPlateau() Schedule too aggressive by default in pytorch?
  
  (Asked by: username_0 on 2024-03-11)
  
  Hi I'm asking about the learning rate schedule [ReduceLROnPlateau()](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html). I've noticed that it decreases the learning rate by a factor of 10 each time and by default the patience is 10. Given phenomena like [double descent](https://en.wikipedia.org/wiki/Double_descent) its default parameters are very concerning.
  
  Not to mention that in practice I find it to be very common that learning might stall for 10 epochs (even if not truly stuck). And for example this LR schedule **(with patience=6)** gives me **learning\_rate=2e-9 after just 100 epochs!!** Presumably it would further decrease it if not for the epsilon parameter, which isn't intuitive as you would think it should decrease gradually...
  
  Are they basing these defaults off of some paper? Can anyone speak to their experience using this schedule & the performance of its default settings? And/or share relevant papers.
  
  **Tags:** learning-rate
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-13. Score: 2 | Accepted answer
  
  I'll start by saying that optimization is as much art as it is science.
  
  [ReduceLROnPlateau](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html) is my favourite scheduler and I've found it performs quite well out of the box. Of course it's good to play with it to achieve optimal results but for what you experience there might be other reasons.
  
  Depending on the size of your data, 100 epochs might be more than enough so this might not be a problem to begin with.
  
  In case this is indeed a problem, since you don't give much details of your training, I can only provide some generic suggestions.
  
  Check if your model learns without a scheduler. Find some good hyperparameters for:
  
  * The starting learning rate. It might be too small to begin with, or even too big (this could mess the network initialization).
  * The batch size cause it could greatly affect training dynamics.
  * Any other parameter you think it's important.
  
  After you have these, then add the scheduler. If you still have issues try one of the following:
  
  * Check if the metric that you're monitoring is the right one, i.e. a loss on a validation set and that it's values make sense.
  * Check the iterations per epoch. A very small number of iterations per epoch could give noisy validation metrics.
  
  > **Comments:**
  > **username_0**: I also like the idea of it, but this is the 2nd time I've found it's defaults problematic because in practice I usually want to train my models for thousands of epochs.  
  > **username_1**: It's hard to help more without knowning the specifics. Consider sharing a bit more information about your model, data and training.  
  > **username_0**: Thanks I think I discovered the problem to be that epochs aren't a proper "unit" because the number of gradient steps that they represent can vary by orders of magnitude depending on the size of the dataset. And these defaults are probably designed for enormous image datasets with many gradient steps.  
  > **username_1**: Good to know you solved it. Yeah these methods target the stochastic gradient descent regime. If you're doing only a small amount of iterations per epoch then measuring the validation error in every epoch can be too frequent, your model doesn't have time to become better (hence noisy validation metrics). It's probably not very efficient either.  
  > **username_0**: Ya your comment about 100 epochs "being more than enough", was enough for me to figure out the rest. Thanks  
  
  ---
  
  