I’m not sure in which situations shuffling might hurt the performance, but a possibility (even though I don’t think it’s highly likely) would be that the non-shuffled training dataset represents the batch statistics better towards the end of the training data and thus the test dataset would see a benefit in better adapted running stats in the batchnorm layers.
This effect can sometimes be observed in time-dependent data, where e.g. a sensor captured the signal and the first part of the data is used as training while the latter would be used as the validation dataset.
Could this be the case for your dataset or are all samples independently sampled?