I think you are confusing the use of confusion matrix.
Suppose you have a classification problem and you get 97% accuracy, but now how do you know if 97% is good or not. Now you use confusion matrix to see the different values. Why is this important? Think of a skewed problem where it is true 99% of the time. In that case getting a accuracy of 99% is not good as you get the same accuracy by predicting the value True everytime.
So you use confusion matrix as a tool to quantify the trust in the accuracy you are getting from your model and depending upon your classification problem you use the numbers from the confusion matrix to further fine-tune or train completely new models.
But you cannot use confusion matrix as your cost function as explained by @rasbt