<p>Oscillating loss is a symptom of divergent training: it can be due to large gradient updates, and/or numerical instability. Moreover, you said that it gets worse with higher LR: that's a confirm.</p>
<p>To fix that you should try:</p>
<ul>
<li>A <strong>decaying lr schedule</strong>, that basically after a given amount of optimization steps (or epochs) your lr is reduced by a factor <span class="math-container">$\gamma$</span> (<span class="math-container">$\gamma &lt; 1$</span>) until a minimum lr is reached. The decay can be <em>linear</em> (so <span class="math-container">$\gamma$</span> is proportional to the number of steps) or <em>exponential</em> for example.</li>
<li><strong>Layer normalization</strong>: this can help to stabilize the gradient, for a faster training.</li>
<li><strong>Gradient clipping</strong>: it can scale the gradients <em>individually</em> (per-layer) or <em>globally</em> to not be above a given norm. Basically it reduces the magnitude of the gradients to control the size of the update.</li>
</ul>
<p>I suggest you to also monitor the (euclidean) norm of both gradients and weights. So that you can understand if gradients are big or exploding/vanishing, and also have a look at possible overfitting.</p>
<p><em><strong>Update</strong></em>: numerical instability of the loss computation can lead to large gradients, and even NaNs. In particular, the suggested usage of the <em>cross-entropy loss</em> is from <strong>logits</strong> (i.e. unnormalized log-probabilities of the classes) since it can be implemented in a numerically stable way, conversely when directly applied to a <code>softmax</code> output it can be unstable.</p>

CC BY-SA 4.0
2023-04-19T06:14:48.320Z
Thanks a lot for your detailed response. I think your suggestions and explanations make total sense. So it could indeed be a learning/gradient related problem due to too strong/big gradients leading to overshooting and alternation. Okay let's assume this is the case, why is it always the same pattern, so better train accuracy/loss in the first 25% of the epoch and worse for the rest? That's the only thing still worrying me.
PatrickSVM
CC BY-SA 4.0
2023-04-19T07:53:45.780Z
Maybe the model after that starts to fit the noise in the data (i.e. starts to overfit, that explains also the lower HR on validation, for blue curve - lr=0.01), the gradients begins unstable and so the loss fluctuates. It can be also a numerical instability in the cross-entropy loss: try computing that from logits (if not doing so). Lastly, you can even try a different (or simpler) optimizer or to tune a bit the Adam's parameters like the betas and epsilon too.
Luca Anzalone
CC BY-SA 4.0
2023-04-19T11:52:43.610Z
Thank you, that makes a lot of sense. I will try and play around with your suggestions!
PatrickSVM
CC BY-SA 4.0
2023-04-18T11:44:26.350Z
2023-04-21T07:15:04.957Z
2023-04-21T07:15:04.957Z
Luca Anzalone
<p>I am training a one-layer unidirectional vanilla GRU on a next item prediction task with regard to the last 10 interacted items. In my original experiment, where I trained on approx. 5.5M samples and validated on around 1M samples, I saw periodic fluctuations in my loss/accuracy curves. Now I made a few experiments with a far smaller dataset (train: 250K, val: 10K) and compared different learning rates. I still saw this effect and I would like to know what causes it, just to understand if it is an issue or not.</p>
<p>I am evaluating every 25% per epoch, so 4 times per epoch. I accumulate the loss and the hit-ratio for the processed training batches and when it is time for evaluation, based on these I compute train metrics and then compute the validation metrics on the entire validation set.</p>
<p>I am using Adam as optimizer.</p>
<p><a href="https://i.sstatic.net/9NtfQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9NtfQ.png" alt="enter image description here" /></a></p>
<p>The plots show HR@10 (=accuracy for top-10 predictions) and CrossEntropyLoss for train and val. One step is referring to 25% of an epoch, so step 0 = 25% of first epoch, step 4 = 25% of second epoch done. You can see that the phenomenon gets more extreme for higher learning rates. The spike in training HR/loss always happens at the first evaluation during an epoch (so 25%). It is paired with a drop in validation accuracy, which gets better for the next three steps (50%, 75% 100% of epoch). Then falls again when spike in training accuracy occurs.</p>
<p>Things I excluded already:</p>
<ul>
<li>data gets fed randomly, it is not always the same order</li>
<li>checked whether gradient is accidentally not reset before calling <code>backward()</code></li>
<li>possible wrong calculation of loss/accuracy due to wrong number of batches I divide by</li>
</ul>
<p>I would appreciate any help to understand what is happening with the model. It still seems to work but I just don't understand what causes this behavior.</p>

<deep-learning><pytorch><loss><validation-loss>
Fluctuations in loss during in epoch evaluation of GRU
PatrickSVM
# Fluctuations in loss during in epoch evaluation of GRU

(Asked by: username_0 on 2023-04-18)

I am training a one-layer unidirectional vanilla GRU on a next item prediction task with regard to the last 10 interacted items. In my original experiment, where I trained on approx. 5.5M samples and validated on around 1M samples, I saw periodic fluctuations in my loss/accuracy curves. Now I made a few experiments with a far smaller dataset (train: 250K, val: 10K) and compared different learning rates. I still saw this effect and I would like to know what causes it, just to understand if it is an issue or not.

I am evaluating every 25% per epoch, so 4 times per epoch. I accumulate the loss and the hit-ratio for the processed training batches and when it is time for evaluation, based on these I compute train metrics and then compute the validation metrics on the entire validation set.

I am using Adam as optimizer.

[![enter image description here](https://i.sstatic.net/9NtfQ.png)](https://i.sstatic.net/9NtfQ.png)

The plots show HR@10 (=accuracy for top-10 predictions) and CrossEntropyLoss for train and val. One step is referring to 25% of an epoch, so step 0 = 25% of first epoch, step 4 = 25% of second epoch done. You can see that the phenomenon gets more extreme for higher learning rates. The spike in training HR/loss always happens at the first evaluation during an epoch (so 25%). It is paired with a drop in validation accuracy, which gets better for the next three steps (50%, 75% 100% of epoch). Then falls again when spike in training accuracy occurs.

Things I excluded already:

* data gets fed randomly, it is not always the same order
* checked whether gradient is accidentally not reset before calling `backward()`
* possible wrong calculation of loss/accuracy due to wrong number of batches I divide by

I would appreciate any help to understand what is happening with the model. It still seems to work but I just don't understand what causes this behavior.

**Tags:** deep-learning, pytorch, loss, validation-loss


---

## 1 answers

### Answer by username_1 on 2023-04-18. Score: 2 | Accepted answer

Oscillating loss is a symptom of divergent training: it can be due to large gradient updates, and/or numerical instability. Moreover, you said that it gets worse with higher LR: that's a confirm.

To fix that you should try:

* A **decaying lr schedule**, that basically after a given amount of optimization steps (or epochs) your lr is reduced by a factor $\gamma$ ($\gamma < 1$) until a minimum lr is reached. The decay can be *linear* (so $\gamma$ is proportional to the number of steps) or *exponential* for example.
* **Layer normalization**: this can help to stabilize the gradient, for a faster training.
* **Gradient clipping**: it can scale the gradients *individually* (per-layer) or *globally* to not be above a given norm. Basically it reduces the magnitude of the gradients to control the size of the update.

I suggest you to also monitor the (euclidean) norm of both gradients and weights. So that you can understand if gradients are big or exploding/vanishing, and also have a look at possible overfitting.

***Update***: numerical instability of the loss computation can lead to large gradients, and even NaNs. In particular, the suggested usage of the *cross-entropy loss* is from **logits** (i.e. unnormalized log-probabilities of the classes) since it can be implemented in a numerically stable way, conversely when directly applied to a `softmax` output it can be unstable.

> **Comments:**
> **username_0**: Thanks a lot for your detailed response. I think your suggestions and explanations make total sense. So it could indeed be a learning/gradient related problem due to too strong/big gradients leading to overshooting and alternation. Okay let's assume this is the case, why is it always the same pattern, so better train accuracy/loss in the first 25% of the epoch and worse for the rest? That's the only thing still worrying me.  
> **username_1**: Maybe the model after that starts to fit the noise in the data (i.e. starts to overfit, that explains also the lower HR on validation, for blue curve - lr=0.01), the gradients begins unstable and so the loss fluctuates. It can be also a numerical instability in the cross-entropy loss: try computing that from logits (if not doing so). Lastly, you can even try a different (or simpler) optimizer or to tune a bit the Adam's parameters like the betas and epsilon too.  
> **username_0**: Thank you, that makes a lot of sense. I will try and play around with your suggestions!  

---

