Thank you @ptrblck  for this solution, it works for me! And thanks @Ghada_Sokar  for bringing this up again.
I could see that calling test_loader causes this issue, without even evaluating the model on test data. However, I did set torch.manual_seed(1) before doing training and testing - but it seems like it is overwritten when test_loader is called, and unless I manually reset it again before training, it works differently. Also, why is the seed updated in every epoch as above (seed+epoch), Is it to ensure that data is randomized differently in each epoch?
Thanks!