Solved by ptrblck in post #2 
              
              
                This might be caused by the shuffling of the data, if you haven’t restored the state of epoch 20 for the data loading before continuing the training. 
If you are shuffling the data, you could try to add 20 “empty” epoch iterations using the DataLoader and check the final performance again.