The BatchNorm layers got updated when you switched back to net.train(True) with the statistics of the test set. After calling net.train(False) again, the BatchNorm is still using the updated running_mean and running_var.
This makes sense, if your training and test set differ in their statistics.