Body: <p>I am training a neural network (specifically a GRU based architecture but I think this is not too relevant for the question). My loss curves, especially the training loss but also the validation loss, show periodic fluctuations and I try to understand why (see picture). I am evaluating the model 4 times per epoch (25%, 50%, 75%, 100%) to monitor training progress, dataset consists of roughly 5.5M sample sequences (the task is item recommendation). As you can see from the picture, the fluctuations are exactly 4 steps long (= one epoch). The metric on the picture is the Hit-Ratio for top-1 predictions so just accuracy on the train set.</p>
  <p><a href="https://i.sstatic.net/xyi13.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xyi13.png" alt="enter image description here" /></a></p>
  <p>The only reason I could think of that would explain these periodic fluctuations would be, that the data is fed to the model in the same order each epoch leading to similar learning signals at similar parts of the epoch. But I load the training data using PyTorchs <code>DataLoader</code> with <code>shuffle=True</code>, so this does not seem to be the reason.</p>
  <p>Can you think of any other reasons for such a behavior?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-05T14:42:32.180Z
Comments.Text: Hi, I am seeing something similar in my training. Do you find out the reason? Thanks!
Comments.UserDisplayName: ch271828n
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-06T16:49:40.847Z
Comments.Text: Hi! I asked this twice in the forum, check out the following link, there I got some good answer I would say. But I could still not 100% pinpoint the issue (or if it is an issue): https://ai.stackexchange.com/questions/40107/fluctuations-in-loss-during-in-epoch-evaluation-of-gru/40109#40109
Comments.UserDisplayName: PatrickSVM
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-07T02:28:27.350Z
Comments.Text: Thank you! That does not fix my issue though :/
Comments.UserDisplayName: ch271828n
Tags: <neural-networks><deep-learning><pytorch><loss>
Title: Periodical fluctuations in loss curves
UserDisplayName: PatrickSVM
ThreadText: # Periodical fluctuations in loss curves
  
  (Asked by: username_0 on 2023-04-11)
  
  I am training a neural network (specifically a GRU based architecture but I think this is not too relevant for the question). My loss curves, especially the training loss but also the validation loss, show periodic fluctuations and I try to understand why (see picture). I am evaluating the model 4 times per epoch (25%, 50%, 75%, 100%) to monitor training progress, dataset consists of roughly 5.5M sample sequences (the task is item recommendation). As you can see from the picture, the fluctuations are exactly 4 steps long (= one epoch). The metric on the picture is the Hit-Ratio for top-1 predictions so just accuracy on the train set.
  
  [![enter image description here](https://i.sstatic.net/xyi13.png)](https://i.sstatic.net/xyi13.png)
  
  The only reason I could think of that would explain these periodic fluctuations would be, that the data is fed to the model in the same order each epoch leading to similar learning signals at similar parts of the epoch. But I load the training data using PyTorchs `DataLoader` with `shuffle=True`, so this does not seem to be the reason.
  
  Can you think of any other reasons for such a behavior?
  
  **Tags:** neural-networks, deep-learning, pytorch, loss
  > **Comments:**
  > **username_1**: Hi, I am seeing something similar in my training. Do you find out the reason? Thanks!  
  > **username_0**: Hi! I asked this twice in the forum, check out the following link, there I got some good answer I would say. But I could still not 100% pinpoint the issue (or if it is an issue): https://ai.stackexchange.com/questions/40107/fluctuations-in-loss-during-in-epoch-evaluation-of-gru/40109#40109  
  > **username_1**: Thank you! That does not fix my issue though :/  
  
  ---
  
  ## 0 answers
  
  