This is probably unrelated, but additionally to the suggested experiment, you could also try to drop the last batch, if it is smaller than the rest by using drop_last=True in your DataLoader.
If the last batch is much smaller than the preceding ones, the running estimates might get a “bad update” , which could yield some unexpected loss values in the beginning of the new epoch.
I doubt this effect is that large, but might be a quick experiment.