Hi,
This is a little strange, because you are getting desirable outputs while you are training the model and only it happens in testing stage. So, literally if you pass test images during trainining but intentionally zero gathered gradients before updating weights, you should get correct predictions. Because of that, I think you might have implemented passing values to test function wrong or just doing something concerning code rather than math/logic.
Try not to use your current code after training stage, and just input a single instance once from training set and another from test set.
sample = # get a single instance batch
model.eval()
with torch.no_grad():
     output = model(sample)


And check if the values are reasonable.
Bests