In case of batchnorm2d and batch size = 1. Does it work for you even in eval() mode? I’m currently using batchnorm2d with batch size = 1, but I have to stay in train() mode, otherwise the accuracy drops dramatically.