@ptrblck actually already mentioned what’s going on there: The BatchNorm statistics are updated when you run in train mode, so you have a different network.
Note that you need to check the state dict or the buffers, not just the parameters.
To abstract the situation: You have something where you get behaviour A or B depending on “the state” but you don’t know what exactly is causing it. It might be natural to actually see where the state, here the state_dict, actually differs. So you’d want to save the state dict before the val run and then load the two and compute the maximum absolute difference or so between the weights for each key. You’ll see that it’s the batch norm statistics.
Best regards
Thomas