I think the main reason is, that you don’t necessarily have batches during testing, which might create bad estimates of the mean and std.
You estimate the statistics of your dataset using the trainset and try to make sure the validation and test data is sampled from the same distribution.
The same would apply e.g. if you use batch norm layers with a batch size of 1.
The running estimates will be noisy and thus the validation and test error might be large.
Relying on a specific batch size seems like a bad idea in the general use case.