this is a newby question I am asking here but for some reason, when I change the batch size at test time, the accuracy of my model changes. Decreasing the batch size reduces the accuracy until a batch size of 1 leads to 11% accuracy although the same model gives me 97% accuracy with a test batch size of 512 (I trained it with batch size 512). I am using a pretrained resnet 50 model and finetuning it on my own images and I am also using .train() and .eval() at train and test times properly. The best reason that I can come up with is that for some reason, the batch normalization layers in the model are still tracking the batch statistics at test time (which they are not supposed to do, instead they should be using the ones saved during the training) because a batch size of 1 should lead to mean(x) = x, and the output of bn layer will become 0, leading to zero prediction at the output and hence 11% accuracy because 11% data is from class 0. Also when I send in the validation and test loaders unshuffled, I get bad numbers but when I shuffle them, the same sets give me +96% accuracy. Can somebody help me please? Thank you so much!
And one more thing, I have only changed the classifier at the end and didn’t use bn there. The rest of the model is the same standard implementation that comes with pytorch models module