Good to hear it’s working, but I would still make sure there is no bug in PyTorch somewhere. 
If you keep the model in model.train() and use e.g. batch norm layers, the running statistics will still be updated.
If you shuffle the test data, these stats should represent the dataset better than a non-shuffled dataset.
Note that this could be considered a data leakage.