You should set your model to evaluation by calling model.eval() in your test method.
This makes sure that some layers like BatchNorm and Dropout are switched to evaluation mode, i.e. for BatchNorm the running statistics are used instead of being updated.
The previously correctly classified samples will have different mean and std values and thus you are updating the BatchNorm layers, which might yield different results.