Answers.Body: <p>The loss graph indicates that the model converged to a local minimum, already after a few epochs, and the weights start to oscillate around it. The learning rate is surely responsible for it, but it's not the only culprit.</p>
  <p>Reducing the learning rate with a scheduler didn't work in your case most likely because SGD applies the same learning rate to all weights, and it seems that the model requires different variation per weight to converge to a better minimum. So, consider trying <a href="https://pytorch.org/docs/stable/generated/torch.optim.Adam.html" rel="nofollow noreferrer">ADAM</a> instead of SGD. By design, ADAM adapts the learning rate differently to each weight, increasing performance a lot compared to SGD, especially for tasks that lead to sparse gradients (looking at your gradients is also a quick check you could perform).</p>
  <p>Other things to keep in mind if ADAM will not work:</p>
  <ul>
  <li>check for mislabeled data (not necessarily manually, a test for it could be also training in subsets of the whole dataset and check how much the performances differ and if the oscillation disappear)</li>
  <li>try a deeper model, if the optimizer still doesn't lead to good results maybe the model is not deep enough, and the oscillations are the results of the model overfitting different batches at every epoch.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T11:50:33.783Z
Answers.LastActivityDate: 2022-01-25T08:59:17.503Z
Answers.LastEditDate: 2022-01-25T08:59:17.503Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>Consider the following loss curve</p>
  <p><a href="https://i.sstatic.net/5QnX2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5QnX2.png" alt="enter image description here" /></a></p>
  <p>The x-axis is the no. of epochs and the y-axis is the loss function.</p>
  <p>You can observe that loss is decreasing drastically for the first few epochs and then starts oscillating.</p>
  <p>The graph given is while training. I am using <a href="https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD" rel="nofollow noreferrer">SGD</a> optimizer. The above graph is for the constant learning rate. I also tested with a scheduler of <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html#:%7E:text=Reduce%20learning%20rate%20when%20a,the%20learning%20rate%20is%20reduced." rel="nofollow noreferrer">decreasing the learning rate on a plateau</a>. But, it is not getting better.</p>
  <p>In general, for this situation, can I safely assume that the learning rate is the only reason? Or can there be any other reasons for this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T09:52:23.857Z
Comments.Text: is it training, validation or test loss? What is the optimizer you choose and did you use a scheduler or constant learning rate?
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T10:20:39.293Z
Comments.Text: @EdoardoGuerriero I updated the details.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-25T09:05:37.360Z
Comments.Text: It might be a good idea to specify 1. the learning rate that you used, 2. the task you were trying to solve, 3. how many epochs the plot corresponds to (i.e. the range of the x-axis that you show us), 4. the range of the y-axis, 5.  the batch size.
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-learning><loss><learning-rate>
Title: Is learning rate the only reason for training loss oscillation after few epochs?
UserDisplayName: hanugm
ThreadText: # Is learning rate the only reason for training loss oscillation after few epochs?
  
  (Asked by: username_0 on 2022-01-24)
  
  Consider the following loss curve
  
  [![enter image description here](https://i.sstatic.net/5QnX2.png)](https://i.sstatic.net/5QnX2.png)
  
  The x-axis is the no. of epochs and the y-axis is the loss function.
  
  You can observe that loss is decreasing drastically for the first few epochs and then starts oscillating.
  
  The graph given is while training. I am using [SGD](https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD) optimizer. The above graph is for the constant learning rate. I also tested with a scheduler of [decreasing the learning rate on a plateau](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html#:%7E:text=Reduce%20learning%20rate%20when%20a,the%20learning%20rate%20is%20reduced.). But, it is not getting better.
  
  In general, for this situation, can I safely assume that the learning rate is the only reason? Or can there be any other reasons for this?
  
  **Tags:** neural-networks, deep-learning, loss, learning-rate
  > **Comments:**
  > **username_1**: is it training, validation or test loss? What is the optimizer you choose and did you use a scheduler or constant learning rate?  
  > **username_0**: @EdoardoGuerriero I updated the details.  
  > **username_2**: It might be a good idea to specify 1. the learning rate that you used, 2. the task you were trying to solve, 3. how many epochs the plot corresponds to (i.e. the range of the x-axis that you show us), 4. the range of the y-axis, 5.  the batch size.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-24. Score: 2
  
  The loss graph indicates that the model converged to a local minimum, already after a few epochs, and the weights start to oscillate around it. The learning rate is surely responsible for it, but it's not the only culprit.
  
  Reducing the learning rate with a scheduler didn't work in your case most likely because SGD applies the same learning rate to all weights, and it seems that the model requires different variation per weight to converge to a better minimum. So, consider trying [ADAM](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html) instead of SGD. By design, ADAM adapts the learning rate differently to each weight, increasing performance a lot compared to SGD, especially for tasks that lead to sparse gradients (looking at your gradients is also a quick check you could perform).
  
  Other things to keep in mind if ADAM will not work:
  
  * check for mislabeled data (not necessarily manually, a test for it could be also training in subsets of the whole dataset and check how much the performances differ and if the oscillation disappear)
  * try a deeper model, if the optimizer still doesn't lead to good results maybe the model is not deep enough, and the oscillations are the results of the model overfitting different batches at every epoch.
  
  
  ---
  
  