Should be alright. Your confusion matrix looks good to me.
You could speed up the code a bit:
model.eval()
conf_mat = torch.zeros(3, 3)
with torch.no_grad():
    for data, target in val_loader:
        ...
        output = model(data)
        _, pred = torch.max(output, dim=1)
        for t, p in zip(target, pred):
            conf_mat[t, p] += 1