Finally, I have figured out the issue lay with the preprocessing of my data - it wasn’t exactly the same at test time as during training. A different model I have trained without batchnorm showed that it wouldn’t work on the test data at all either. I find it quite interesting that the model with batchnorm is actually able to perform quite well in train() mode, even though the normalization is different, causing the model to fail in eval() mode.
Thanks for the pointer to GroupNorm, I haven’t been aware of it but will surely check it out!