Based on your description it seems that the running stats in batchnorm layers are not able to represent the mean and std of the activations for the validation dataset.
This could happen, if the training and validation datasets do not have similar statistics, which could happen by e.g. a bad dataset split, datasets sampled from different domains, missing preprocessing in the validation dataset etc.
You could try to change the momentum of the batchnorm layers and see, if the stats would represent the activations better. If you remove the internal stats I guess that the batch stats would be used, so it’s expected that the results would thus depend on the batch size.