Usually you would calculate the validation loss and accuracy after each epoch and could also store the predictions for the entire validation dataset and calculate the confusion matrix afterwards.
If you want to only compute the confusion matrix for the “best” model model, store the state_dict of the model if the validation loss decreases (or the validation accuracy increases), reload the state_dict after the training is finished, and calculate the confusion matrix afterwards.