Solved by smth in post #2 
              
              
                i dont think this is because of momentum. It is probably because of the way new samples are selected in the dataset. 
PyTorch selects samples from dataset without replacement. 
Which means, at the beginning of a new epoch, it is likely that you saw a sample in training set that you saw at the end of…