making a cost function out of a confusion matrix doesn’t make sense to me – how do you differentiate that? It may be useful as a weighting term though. You can check into classification cost matrix: I.e.,  you can add higher weights for certain classes based on the confusion matrix cells. By default, your classification cost matrix is a diagonal matrix of 0’s.