Answers.Body: <p>There are evidences that neural networks trained on a given dataset (or domain) like MNIST, wrongly put some probability mass on datasets from a completely different domain, i.e. the out-of-distribution (OOD) data.</p>
  <p>This is an open research topic that is also related to <em>anomaly</em> and <em>novelty detection</em>, and as well as <em>out-of-domain generalization</em>: the latter about correctly predicting OOD data even without training on them.</p>
  <p>In your case, what you want to do is to constrain the classifier to learn a representation of each digit that is <em>disjoint</em> (has no support) w.r.t. OOD data. There is not yet an universal recipe for this, so I'll write some ideas:</p>
  <ol>
  <li>As suggested by @user253751 you can pick some OOD data, like <em>fashion-MNIST</em> and <em>CIFAR-10</em>, and add an extra <strong>non-digit class</strong> and train your classifier again. This is a very simple approach that might improve OOD performance a bit.</li>
  <li>A more involved approach would make use of some <em>representation learning</em> technique, like <strong>self-supervised contrastive learning</strong> (e.g. <a href="https://keras.io/examples/vision/simsiam/" rel="nofollow noreferrer">SimSiam</a>). In this case you use <em>data-augmentation</em> to enable the model better understand what a digit is. Indeed, I think that combining this method with approach 1, should be quite effective.</li>
  <li>You can use <strong>advanced data-augmentations</strong> like <a href="https://arxiv.org/pdf/1905.04899.pdf" rel="nofollow noreferrer">CutMix</a> and <a href="https://arxiv.org/pdf/1710.09412.pdf" rel="nofollow noreferrer">MixUp</a> to further regularize your model, by explicitly training it in the &quot;interpolation region&quot; between classes. This can be either applied to a MNIST-only training, as well method 1 and 2.</li>
  <li><strong>AE + classifier:</strong> first train an auto-encoder (AE: see <a href="https://keras.io/examples/vision/autoencoder/" rel="nofollow noreferrer">this</a> and <a href="https://keras.io/examples/generative/vae/" rel="nofollow noreferrer">VAEs</a>) to reconstruct the digits, then you train a classifier from its latent space. The AE should have learned a compact representation of the digits, so train a classifier from that representation may be helpful. This is also reasonably simple to implement.</li>
  <li>In the same spirit of 2, there are more complicated methods that involve <strong>clustering</strong>, like <a href="http://arxiv.org/abs/1511.06335" rel="nofollow noreferrer">deep embedding clustering</a>.</li>
  <li><strong>Bayesian NN</strong>: I elaborate a bit on this since you mentioned it. A BNN learns a  probability distribution over weights (doing this per-layer), meaning that each time you forward the network a different realization of the weights is sampled. You can exploit the variance of the predictions (due to sampled weights) to sort of estimate a confidence score for each sample, a part from the predicted class. In principle, you can accept only samples with a minimum confidence level and reject the others. I don't know how this would perform, but you can try it as well. (In case you use <code>tensorflow</code>, I suggest you to use <a href="https://www.tensorflow.org/probability" rel="nofollow noreferrer">tensorflow-probability</a> for easy BNN implementation.)</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T11:13:36.297Z
Answers.Text: I want to add another approach: GANs are typically trained in a fashion where one side produces an image of a digit, and the other side has to decide whether a given sample is real or made up. Doing this with the MNIST dataset might result in a discriminator that is able to tell digits from non-digits.
Answers.UserDisplayName: N. Kiefer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T17:27:47.447Z
Answers.Text: @N.Kiefer True, but actually the GAN's generator is trained by fooling the discriminator. Moreover, at least theoretically, the optimum (i.e. equilibrium of the min-max game between generator and discriminator) is achieved when the discriminator outputs 0.5 probability for both fake and true data. This means the generator is high quality, but that the discriminator is also unusable too.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-25T21:04:42.360Z
Answers.LastActivityDate: 2023-04-25T21:04:42.360Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I've done an MNIST digit recognition neural network.</p>
  <p>When you put images in that are completely unlike its training data, it still tries to classify them as digits. Sometimes it strongly classifies nonsense data as being a specific digit.</p>
  <p>I am interested in the problem of rejecting nonsense inputs generally, and want a solution that is effective for MNIST in particular. I think the most basic approach is just to have a confidence threshold for the standard network. I have also heard about a Bayesian approach.</p>
  <p>I have read that in high dimensional vectors like these bitmap images &quot;almost all data is extrapolation&quot;, I think this makes rejection of out of distribution a difficult problem, but I feel like it is an important problem too so I will appreciate information on this. Thank you.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-24T21:50:12.067Z
Comments.Text: have you considered adding a not-a-digit class and adding a bunch of not-a-digit examples to the training set?
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <probability-distribution><mnist>
Title: How can I make an MNIST digit recognizer that rejects out-of-distribution data?
UserDisplayName: river
ThreadText: # How can I make an MNIST digit recognizer that rejects out-of-distribution data?
  
  (Asked by: username_0 on 2023-04-24)
  
  I've done an MNIST digit recognition neural network.
  
  When you put images in that are completely unlike its training data, it still tries to classify them as digits. Sometimes it strongly classifies nonsense data as being a specific digit.
  
  I am interested in the problem of rejecting nonsense inputs generally, and want a solution that is effective for MNIST in particular. I think the most basic approach is just to have a confidence threshold for the standard network. I have also heard about a Bayesian approach.
  
  I have read that in high dimensional vectors like these bitmap images "almost all data is extrapolation", I think this makes rejection of out of distribution a difficult problem, but I feel like it is an important problem too so I will appreciate information on this. Thank you.
  
  **Tags:** probability-distribution, mnist
  
  > **Comments:**
  > **username_1**: have you considered adding a not-a-digit class and adding a bunch of not-a-digit examples to the training set?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-04-25. Score: 2
  
  There are evidences that neural networks trained on a given dataset (or domain) like MNIST, wrongly put some probability mass on datasets from a completely different domain, i.e. the out-of-distribution (OOD) data.
  
  This is an open research topic that is also related to *anomaly* and *novelty detection*, and as well as *out-of-domain generalization*: the latter about correctly predicting OOD data even without training on them.
  
  In your case, what you want to do is to constrain the classifier to learn a representation of each digit that is *disjoint* (has no support) w.r.t. OOD data. There is not yet an universal recipe for this, so I'll write some ideas:
  
  1. As suggested by @user253751 you can pick some OOD data, like *fashion-MNIST* and *CIFAR-10*, and add an extra **non-digit class** and train your classifier again. This is a very simple approach that might improve OOD performance a bit.
  2. A more involved approach would make use of some *representation learning* technique, like **self-supervised contrastive learning** (e.g. [SimSiam](https://keras.io/examples/vision/simsiam/)). In this case you use *data-augmentation* to enable the model better understand what a digit is. Indeed, I think that combining this method with approach 1, should be quite effective.
  3. You can use **advanced data-augmentations** like [CutMix](https://arxiv.org/pdf/1905.04899.pdf) and [MixUp](https://arxiv.org/pdf/1710.09412.pdf) to further regularize your model, by explicitly training it in the "interpolation region" between classes. This can be either applied to a MNIST-only training, as well method 1 and 2.
  4. **AE + classifier:** first train an auto-encoder (AE: see [this](https://keras.io/examples/vision/autoencoder/) and [VAEs](https://keras.io/examples/generative/vae/)) to reconstruct the digits, then you train a classifier from its latent space. The AE should have learned a compact representation of the digits, so train a classifier from that representation may be helpful. This is also reasonably simple to implement.
  5. In the same spirit of 2, there are more complicated methods that involve **clustering**, like [deep embedding clustering](http://arxiv.org/abs/1511.06335).
  6. **Bayesian NN**: I elaborate a bit on this since you mentioned it. A BNN learns a probability distribution over weights (doing this per-layer), meaning that each time you forward the network a different realization of the weights is sampled. You can exploit the variance of the predictions (due to sampled weights) to sort of estimate a confidence score for each sample, a part from the predicted class. In principle, you can accept only samples with a minimum confidence level and reject the others. I don't know how this would perform, but you can try it as well. (In case you use `tensorflow`, I suggest you to use [tensorflow-probability](https://www.tensorflow.org/probability) for easy BNN implementation.)
  
  > **Comments:**
  > **username_3**: I want to add another approach: GANs are typically trained in a fashion where one side produces an image of a digit, and the other side has to decide whether a given sample is real or made up. Doing this with the MNIST dataset might result in a discriminator that is able to tell digits from non-digits.  
  > **username_2**: @N.username_3 True, but actually the GAN's generator is trained by fooling the discriminator. Moreover, at least theoretically, the optimum (i.e. equilibrium of the min-max game between generator and discriminator) is achieved when the discriminator outputs 0.5 probability for both fake and true data. This means the generator is high quality, but that the discriminator is also unusable too.  
  
  ---
  
  