Answers.Body: <blockquote>
    <p>I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?</p>
  </blockquote>
  
  <p>Really you should not, because the symbols used (Arabic numerals) do not have direct relation to quantity in the same way e.g. tally counts or dots do. They are good candidates for classification, and despite the conventional mapping to quantity when you read them, the symbols themselves are poor candidates for regression, because for instance the symbols <span class="math-container">$3$</span> and <span class="math-container">$4$</span> do not differ in a way that captures quantity in any intuitive manner. </p>
  
  <p>However, if you are keen to do this, it is relatively simple to construct a suitable loss function in most auto-differentiating frameworks. You will need to read up on how to do so. For instance, <a href="https://stackoverflow.com/a/43821374/2001031">here is a Stack Overflow answer explaining where to start with writing custom loss function in Keras</a>.</p>
  
  <p>In order for your loss function to work, it will need to be differentiable and  smoothly changing as predictions get better. That rules out using any form of <code>argmax</code> for the current prediction. If you want to stick with softmax for the final layer, then I suggest using a mean squared error against the <em>expected</em> prediction, e.g. if <span class="math-container">$d_i$</span> is the numerical digit for example <span class="math-container">$i$</span> and <span class="math-container">$y_{i,j}$</span> is the ground truth expressed as a one-hot vector, where <span class="math-container">$i$</span> is the example and <span class="math-container">$j$</span> is the digit class, then <span class="math-container">$\hat{y}_{i,j}$</span> is the probability predicted by your model. You could use <span class="math-container">$\hat{d}_i = \sum_{j=0}^9 j\hat{y}_{i,j}$</span> for the expected value and MSE loss of <span class="math-container">$\mathcal{L}(d_i,\hat{d}_i) = \frac{1}{2}(\hat{d}_i - d_i)^2$</span></p>
  
  <p>You can also use a weighted sum of the MSE loss and cross entropy loss as your final loss, with the balance between the two losses being a new hyperparameter of your model. </p>
  
  <p>Note this solution makes <span class="math-container">$0$</span> close to <span class="math-container">$1$</span> but far away from <span class="math-container">$9$</span>. If you want the digits to be considered close on a cycle (e.g. <span class="math-container">$8$</span> is closer to <span class="math-container">$1$</span> than it is to <span class="math-container">$4$</span>) the you will need something more creative.</p>
  
  <p>Whilst I don't think this will help you discover any improvements to MNIST classification, combining two or more loss functions to achieve a more complex goal can be really useful sometimes, so it is a skill worth practicing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-16T21:00:29.850Z
Answers.LastActivityDate: 2020-02-16T21:13:23.413Z
Answers.LastEditDate: 2020-02-16T21:13:23.413Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am making an MNIST classifier. I am using categorical cross-entropy as my loss function. I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?</p>
  
Tags: <machine-learning><classification><objective-functions><cross-entropy><categorical-crossentropy>
Title: How should I penalize the model proportionally to the error?
UserDisplayName: Josh Goldman
ThreadText: # How should I penalize the model proportionally to the error?
  
  (Asked by: username_0 on 2020-02-16)
  
  I am making an MNIST classifier. I am using categorical cross-entropy as my loss function. I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?
  
  **Tags:** machine-learning, classification, objective-functions, cross-entropy, categorical-crossentropy
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-16. Score: 1 | Accepted answer
  
  > I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?
  
  Really you should not, because the symbols used (Arabic numerals) do not have direct relation to quantity in the same way e.g. tally counts or dots do. They are good candidates for classification, and despite the conventional mapping to quantity when you read them, the symbols themselves are poor candidates for regression, because for instance the symbols $3$ and $4$ do not differ in a way that captures quantity in any intuitive manner.
  
  However, if you are keen to do this, it is relatively simple to construct a suitable loss function in most auto-differentiating frameworks. You will need to read up on how to do so. For instance, [here is a Stack Overflow answer explaining where to start with writing custom loss function in Keras](https://stackoverflow.com/a/43821374/2001031).
  
  In order for your loss function to work, it will need to be differentiable and smoothly changing as predictions get better. That rules out using any form of `argmax` for the current prediction. If you want to stick with softmax for the final layer, then I suggest using a mean squared error against the *expected* prediction, e.g. if $d\_i$ is the numerical digit for example $i$ and $y\_{i,j}$ is the ground truth expressed as a one-hot vector, where $i$ is the example and $j$ is the digit class, then $\hat{y}\_{i,j}$ is the probability predicted by your model. You could use $\hat{d}\_i = \sum\_{j=0}^9 j\hat{y}\_{i,j}$ for the expected value and MSE loss of $\mathcal{L}(d\_i,\hat{d}\_i) = \frac{1}{2}(\hat{d}\_i - d\_i)^2$
  
  You can also use a weighted sum of the MSE loss and cross entropy loss as your final loss, with the balance between the two losses being a new hyperparameter of your model.
  
  Note this solution makes $0$ close to $1$ but far away from $9$. If you want the digits to be considered close on a cycle (e.g. $8$ is closer to $1$ than it is to $4$) the you will need something more creative.
  
  Whilst I don't think this will help you discover any improvements to MNIST classification, combining two or more loss functions to achieve a more complex goal can be really useful sometimes, so it is a skill worth practicing.
  
  
  ---
  
  