I have observed the same loss pattern on an automatic speech recognition task using the default PyTorch dataloader (sampling without replacement). Applying instead the with-replacement random sampler suggested by @smth removes the pattern, as would be expected.
I am aware of results in the literature indicating that sampling without replacement may yield faster convergence on some problems (https://arxiv.org/abs/1603.00570 13, https://arxiv.org/abs/1202.4184v1 3). However, I feel that the question remains as to why we would be okay with the loss pattern induced by sampling without replacement?
Drops at the interface between two epochs could be expected simply due to seeing samples which have already been fitted again for the first (or second or third or …) time. This could indicate that we are indeed learning to better classify that specific sample. However, the reason for the increasing trend in the loss during a single epoch evades me.
One line of thought is that fitting to the examples in the first batch of an epoch worsens the model performance on any other examples in the training set; and so forth for all batches of an epoch. However, wouldn’t this be expected to result in an increasing validation loss? Aren’t we in a sense overfitting to each training example during the epoch?
Thanks in advance.