Solved by Yelnat10 in post #3 
              
              
                Hey, sorry bout that, Forgot I made this post (and then subsequently my password). The way I’m implementing my cost matrix is not by building it into the loss function but instead by using the method defined in Bishops book “Pattern Recognition And Machine Learning” 
Bishop gives us this formula to …