Solved by ptrblck in post #2 
              
              
                You should set your model to evaluation by calling model.eval() in your test method. 
This makes sure that some layers like BatchNorm and Dropout are switched to evaluation mode, i.e. for BatchNorm the running statistics are used instead of being updated. 
The previously correctly classified samples…