BatchNorm uses it’s running statistics (running_mean and running_var) without updating it in eval mode.
Usually, it shouldn’t be a problem using a batch size of 1, if you set your model to eval.
It’s a bit strange however, that you got better results keeping the model in train, since the BatchNorm layers will be updated with the “batch statistics” of your single input.
How large is the discrepancy between the training and test accuracy?