This will make sure to use the running stats on your test samples instead of calculating them from the current batch.
You might play a bit with the momentum, which might help, if your batch stats are quite different.
BatchNorm layers usually work good with a batch size of > 64 so you might try group normalization.