I think you could create the confusion matrix with sklearn (or calculate it manually), create a plot with matplotlib and send the image to tensorboard.
I’m not sure, if tensorboard can directly display this kind of data.