Hello. Sorry for the delay and thanks for your reply. I kept training and valuating the network without the model.eval() mode. Even though, when I the model to run with the test set the same problem happens.
I tried to run the BN layers in eval mode during training and the performance during training is also degraded.
I saw some modifications on track_running_stats, setting it to False. But it didn’t work for me.
Apparently it has to do with the PyTorch’s BN implementation. But I still don’t understand why this problem is happening.