Answers.Body: <p>Oscillating loss is a symptom of divergent training: it can be due to large gradient updates, and/or numerical instability. Moreover, you said that it gets worse with higher LR: that's a confirm.</p>
  <p>To fix that you should try:</p>
  <ul>
  <li>A <strong>decaying lr schedule</strong>, that basically after a given amount of optimization steps (or epochs) your lr is reduced by a factor <span class="math-container">$\gamma$</span> (<span class="math-container">$\gamma &lt; 1$</span>) until a minimum lr is reached. The decay can be <em>linear</em> (so <span class="math-container">$\gamma$</span> is proportional to the number of steps) or <em>exponential</em> for example.</li>
  <li><strong>Layer normalization</strong>: this can help to stabilize the gradient, for a faster training.</li>
  <li><strong>Gradient clipping</strong>: it can scale the gradients <em>individually</em> (per-layer) or <em>globally</em> to not be above a given norm. Basically it reduces the magnitude of the gradients to control the size of the update.</li>
  </ul>
  <p>I suggest you to also monitor the (euclidean) norm of both gradients and weights. So that you can understand if gradients are big or exploding/vanishing, and also have a look at possible overfitting.</p>
  <p><em><strong>Update</strong></em>: numerical instability of the loss computation can lead to large gradients, and even NaNs. In particular, the suggested usage of the <em>cross-entropy loss</em> is from <strong>logits</strong> (i.e. unnormalized log-probabilities of the classes) since it can be implemented in a numerically stable way, conversely when directly applied to a <code>softmax</code> output it can be unstable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-19T06:14:48.320Z
Answers.Text: Thanks a lot for your detailed response. I think your suggestions and explanations make total sense. So it could indeed be a learning/gradient related problem due to too strong/big gradients leading to overshooting and alternation. Okay let's assume this is the case, why is it always the same pattern, so better train accuracy/loss in the first 25% of the epoch and worse for the rest? That's the only thing still worrying me.
Answers.UserDisplayName: PatrickSVM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-19T07:53:45.780Z
Answers.Text: Maybe the model after that starts to fit the noise in the data (i.e. starts to overfit, that explains also the lower HR on validation, for blue curve - lr=0.01), the gradients begins unstable and so the loss fluctuates. It can be also a numerical instability in the cross-entropy loss: try computing that from logits (if not doing so). Lastly, you can even try a different (or simpler) optimizer or to tune a bit the Adam's parameters like the betas and epsilon too.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-19T11:52:43.610Z
Answers.Text: Thank you, that makes a lot of sense. I will try and play around with your suggestions!
Answers.UserDisplayName: PatrickSVM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-18T11:44:26.350Z
Answers.LastActivityDate: 2023-04-21T07:15:04.957Z
Answers.LastEditDate: 2023-04-21T07:15:04.957Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am training a one-layer unidirectional vanilla GRU on a next item prediction task with regard to the last 10 interacted items. In my original experiment, where I trained on approx. 5.5M samples and validated on around 1M samples, I saw periodic fluctuations in my loss/accuracy curves. Now I made a few experiments with a far smaller dataset (train: 250K, val: 10K) and compared different learning rates. I still saw this effect and I would like to know what causes it, just to understand if it is an issue or not.</p>
  <p>I am evaluating every 25% per epoch, so 4 times per epoch. I accumulate the loss and the hit-ratio for the processed training batches and when it is time for evaluation, based on these I compute train metrics and then compute the validation metrics on the entire validation set.</p>
  <p>I am using Adam as optimizer.</p>
  <p><a href="https://i.sstatic.net/9NtfQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9NtfQ.png" alt="enter image description here" /></a></p>
  <p>The plots show HR@10 (=accuracy for top-10 predictions) and CrossEntropyLoss for train and val. One step is referring to 25% of an epoch, so step 0 = 25% of first epoch, step 4 = 25% of second epoch done. You can see that the phenomenon gets more extreme for higher learning rates. The spike in training HR/loss always happens at the first evaluation during an epoch (so 25%). It is paired with a drop in validation accuracy, which gets better for the next three steps (50%, 75% 100% of epoch). Then falls again when spike in training accuracy occurs.</p>
  <p>Things I excluded already:</p>
  <ul>
  <li>data gets fed randomly, it is not always the same order</li>
  <li>checked whether gradient is accidentally not reset before calling <code>backward()</code></li>
  <li>possible wrong calculation of loss/accuracy due to wrong number of batches I divide by</li>
  </ul>
  <p>I would appreciate any help to understand what is happening with the model. It still seems to work but I just don't understand what causes this behavior.</p>
  
Tags: <deep-learning><pytorch><loss><validation-loss>
Title: Fluctuations in loss during in epoch evaluation of GRU
UserDisplayName: PatrickSVM
ThreadText: # Fluctuations in loss during in epoch evaluation of GRU
  
  (Asked by: username_0 on 2023-04-18)
  
  I am training a one-layer unidirectional vanilla GRU on a next item prediction task with regard to the last 10 interacted items. In my original experiment, where I trained on approx. 5.5M samples and validated on around 1M samples, I saw periodic fluctuations in my loss/accuracy curves. Now I made a few experiments with a far smaller dataset (train: 250K, val: 10K) and compared different learning rates. I still saw this effect and I would like to know what causes it, just to understand if it is an issue or not.
  
  I am evaluating every 25% per epoch, so 4 times per epoch. I accumulate the loss and the hit-ratio for the processed training batches and when it is time for evaluation, based on these I compute train metrics and then compute the validation metrics on the entire validation set.
  
  I am using Adam as optimizer.
  
  [![enter image description here](https://i.sstatic.net/9NtfQ.png)](https://i.sstatic.net/9NtfQ.png)
  
  The plots show HR@10 (=accuracy for top-10 predictions) and CrossEntropyLoss for train and val. One step is referring to 25% of an epoch, so step 0 = 25% of first epoch, step 4 = 25% of second epoch done. You can see that the phenomenon gets more extreme for higher learning rates. The spike in training HR/loss always happens at the first evaluation during an epoch (so 25%). It is paired with a drop in validation accuracy, which gets better for the next three steps (50%, 75% 100% of epoch). Then falls again when spike in training accuracy occurs.
  
  Things I excluded already:
  
  * data gets fed randomly, it is not always the same order
  * checked whether gradient is accidentally not reset before calling `backward()`
  * possible wrong calculation of loss/accuracy due to wrong number of batches I divide by
  
  I would appreciate any help to understand what is happening with the model. It still seems to work but I just don't understand what causes this behavior.
  
  **Tags:** deep-learning, pytorch, loss, validation-loss
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-18. Score: 2 | Accepted answer
  
  Oscillating loss is a symptom of divergent training: it can be due to large gradient updates, and/or numerical instability. Moreover, you said that it gets worse with higher LR: that's a confirm.
  
  To fix that you should try:
  
  * A **decaying lr schedule**, that basically after a given amount of optimization steps (or epochs) your lr is reduced by a factor $\gamma$ ($\gamma < 1$) until a minimum lr is reached. The decay can be *linear* (so $\gamma$ is proportional to the number of steps) or *exponential* for example.
  * **Layer normalization**: this can help to stabilize the gradient, for a faster training.
  * **Gradient clipping**: it can scale the gradients *individually* (per-layer) or *globally* to not be above a given norm. Basically it reduces the magnitude of the gradients to control the size of the update.
  
  I suggest you to also monitor the (euclidean) norm of both gradients and weights. So that you can understand if gradients are big or exploding/vanishing, and also have a look at possible overfitting.
  
  ***Update***: numerical instability of the loss computation can lead to large gradients, and even NaNs. In particular, the suggested usage of the *cross-entropy loss* is from **logits** (i.e. unnormalized log-probabilities of the classes) since it can be implemented in a numerically stable way, conversely when directly applied to a `softmax` output it can be unstable.
  
  > **Comments:**
  > **username_0**: Thanks a lot for your detailed response. I think your suggestions and explanations make total sense. So it could indeed be a learning/gradient related problem due to too strong/big gradients leading to overshooting and alternation. Okay let's assume this is the case, why is it always the same pattern, so better train accuracy/loss in the first 25% of the epoch and worse for the rest? That's the only thing still worrying me.  
  > **username_1**: Maybe the model after that starts to fit the noise in the data (i.e. starts to overfit, that explains also the lower HR on validation, for blue curve - lr=0.01), the gradients begins unstable and so the loss fluctuates. It can be also a numerical instability in the cross-entropy loss: try computing that from logits (if not doing so). Lastly, you can even try a different (or simpler) optimizer or to tune a bit the Adam's parameters like the betas and epsilon too.  
  > **username_0**: Thank you, that makes a lot of sense. I will try and play around with your suggestions!  
  
  ---
  
  