Hey ! I got the same issue on my code. The results are very bad with a batch size of 1, which is not very practical for evaluation of a single image.
I put model.eval() as well as
    for child in model.children():
            if type(child) == nn.BatchNorm2d:
                child.track_running_stats = False

EDIT: does eval makes the use of the running stats but do not save the variance and mean ? Are the variance and mean supposed to scale with the batch size ?
Any ideas ? The network is a stock ResNet adapted for regression instead of classification