I am trying to make a face recognition model with PyTorch. My model performs well with both loss scores for training and validation close to 0. The problem is when I tested it using same input, it gives me different results like below:

and below

The left side is the true label and the right one is the prediction output. I set random seed and ran

model.eval()

before feeding the input to the model.
Does anyone know how to solve this?
Please advise