This is very likely to be caused by the BatchNorm layer. BatchNorm computes a running mean and variance that is used during prediction.
For some reason, the running mean/variance are not a good approximation of the true distribution of the activations during prediction. Maybe the batch size you used was to small, maybe the default momentum parameter for the running mean/variance (0.1 IIRC) for BatchNorm is too large in your case.