Hi. If I understand your problem correctly, you are getting a slightly different confusion matrix values each time you run the complete cell that makes the inference using your trained model.
If my understanding is correct, then one thing to try out in this case would be to set seed, something like this:
seed = 0
torch.manual_seed(seed)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(seed)

Setting a seed helps in result reproducibility. You can read more on this here: Reproducibility — PyTorch 1.7.0 documentation