The following is only my opinion, which is developed from various articles I saw on the internet and my speculations, so please correct me if u find a better explanation:

If the batch size is 1, batch norm is bad because batch norm requires a relative big batch size to be able to function well.
If the batch size is bigger, there should be some padding values for sure, and batch norm will take that into account, which will probably degrade the performance.