Could you be providing the same labels to the confusion matrix plotting function, like plotting predictions versus predictions or ground truth versus ground truth? That kind of mistake is one that I make all the time.