I had model.eval() in my code but the issue still exists.
I tried to reproduce the issue on a very small architecture on MNIST. Here is the link: https://colab.research.google.com/drive/1A6Ey6Z6mhUBWBP5Y-EwNIIFyg_q6JGkQ?usp=sharing
When test() is called at the end of the training only, accuracy 9508/10000
When test() is called during the training as well, accuracy: 9484/10000