Do your datasets come from different domains? Sometimes it helps to keep the model in train, pass some validation data through it to update the BatchNorm layers without backpropagating, and then switch to eval. This is used in some GANs.
However, you should be careful about the error estimate, since the validation data is not completely “clean” anymore in my opinion.
How large is the difference between the training and validation error?