Recently I have found an interesting behavior (may be I am doing something wrong…) during the ResNet training.
When I apply model.eval() [with torch.no_grad()] and evaluate on validation set and switch back to model.train(), the training loss increases suddenly (gets significantly bigger than the previous training losses). But this the case for the first model.eval() only…
I was wondering if model.eval() resets BN statistics (moving mean and moving variance)?