Answers.Body: <p>Your classifier is specifically learning the ways in which <strong>0</strong>s are different from other digits, 
  <strong>not</strong> what it really means for a digit to <strong>be</strong> a zero.</p>
  
  <p>Philosophically, you could say the model appears to have some powerful understanding when restricted
  to a tightly controlled domain, but that facade is lifted as soon as you throw any sort of wrench in 
  the works.</p>
  
  <p>Mathematically, you could say that the model is simply optimizing a classification metric for data
  drawn from a specific distribution, and when you give it data from a different distribution, all
  bets are off.</p>
  
  <p>The go-to answer is to collect or generate data like the data you expect the model to deal with
  (in practice, the effort required to do so can vary dramatically depending upon the application). In
  this case, that could involve drawing a bunch of random scribbles and adding them to your training
  data set. At this point you must ask, now how do I label them? You will want a new "other" or 
  "non-digit" class so that your model can learn to categorize these scribbles separately from digits.
  After retraining, your model should now better deal with these cases.</p>
  
  <p>However, you may then ask, but what if I gave it color images of digits?  Or color images of farm
  animals?  Maybe pigs will be classified as zeros because they are round.  This problem is a
  fundamental property of the way deep learning is orchestrated.  Your model is not capable of higher
  order logic, which means it can seem to go from being very intelligent to very dumb by just throwing
  the slightest curve ball at it.  For now, all deep learning does is recognize patterns in data that
  allow it to minimize some loss function.</p>
  
  <p>Deep learning is a fantastic tool, but not an all-powerful omnitool.  Bear in mind its limitations
  and use it where appropriate, and it will serve you well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T20:48:20.590Z
Answers.Text: @Fnguyen yes totally agree there. Why wouldn't it at least think that maybe it's looking at something else. Although I'm not really sure how overfitting would cause this issue. Unless there were many zeros in the training set that looked like this image. For reference I used MNIST, then post trained on a set of 3000 digits with slightly different handwriting styles (turns out South Koreans write digits a bit different than westeners).
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T02:29:32.893Z
Answers.LastActivityDate: 2020-01-29T02:29:32.893Z
Answers.UserDisplayName: Apollys supports Monica
Answers.Body: <p>Apollys,</p>
  
  <p>That's a very well thought out response. Particularly, the philosophical  discussion of the essence of "0-ness."</p>
  
  <p>I haven't actually performed this experiment, so caveat emptor... I wonder how well an "other" class would actually work. The ways in which "other" differs from "digit" has infinite variability (or at least its only limitation is the cardinality of the input layer).</p>
  
  <p>The NN decides whether something is more of one class or more of a different class. If there isn't an essence in common among other "non-digits", I don't believe it will do well at identifying "other" as the catch-all for everything that has low confidence level of classification.</p>
  
  <p>This approach still doesn't identify what it is to <strong><em>be</em></strong> "not-digit". It identifies how all the things that are "other" differ from the other labeled inputs -- probably poorly, depending on the variability of the "non-digit" labeled data. (i.e. is it numerically exhaustive, many times over, of all random scribbles?) Thoughts?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T20:08:15.780Z
Answers.Text: This answer looks like it should be a comment, but it looks like you don't have enough reputation to comment. You brought up some great concerns about the effectiveness of adding an "other" class. It can be really hard to predict what will or won't work well with neural networks.
Answers.UserDisplayName: Kyle A
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T20:50:28.897Z
Answers.Text: Totally agree here. I was thinking something along the lines when I saw the suggestion. @MatiasValdenegro (don't know if tagging you here works), I was wondering what your thoughts on such an approach might be.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-30T00:17:05.690Z
Answers.Text: Yes, this is an important point.  Once again, the "other" class will have to come from some specific distribution and have enough samples to cover that distribution.  That means if you really want your model to learn what it means to be a digit rather than anything else that is possible, you will need a (probably intractably) **lot** of data.  In this way, we see that deep learning is largely about understanding and confining the problem domain very well.
Answers.UserDisplayName: Apollys supports Monica
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-30T09:30:54.733Z
Answers.Text: Point from practical experience: having _one_ other class may work, but in some cases it can help to have multiple. I'm not aware of a method to determine up front what the best number of "other" classes is.
Answers.UserDisplayName: MSalters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-31T15:14:29.173Z
Answers.Text: An approach may be to use/train some form of binary classifier to discard non-digit examples prior to the recognition step (i.e., 0-9 classification). You'll have to maximise recall of the digit inputs, so there may still be a few non-digits slip through, but such an approach may reduce the detriment to the performance of classifying the real digits.
Answers.UserDisplayName: mrkwse
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T14:07:54.923Z
Answers.LastActivityDate: 2020-01-29T14:07:54.923Z
Answers.UserDisplayName: Rich Chase
Answers.Body: <h2>Broken assumptions</h2>
  
  <p>Generalization relies on making strong assumptions (no free lunch, etc). If you break your assumptions, then you're not going to have a good time. A key assumption of a standard digit-recognition classifier like MNIST is that you're classifying pictures that actually contain a single digit. If your real data contains pictures that have non-digits, then that means that your real data is <em>not</em> similar to training data but is conceptually very, very different.</p>
  
  <p>If that's a problem (as in this case) then one way to treat that is to explicitly break that assumption and train a model that not only recognizes digits 0-9 but also recognizes whether there's a digit at all, and is able to provide an answer "that's not a digit", so a 11-class classifier instead of a 10-class one. MNIST training data is not sufficient for that, but you can use some kind of 'distractor' data to provide the not-a-digit examples. For example, you could use some dataset of letters (perhaps omitting I, l, O and B) transformed to look similar to MNIST data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T01:48:25.913Z
Answers.LastActivityDate: 2020-01-29T01:48:25.913Z
Answers.UserDisplayName: Peteris
Answers.Body: <p>I'm an amateur with neural networks, but I will illustrate my understanding of how this problem comes to be. </p>
  
  <p>First, lets see how trivial neural network classifies 2D input into two classes : 
  <a href="https://i.sstatic.net/OgldC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OgldC.png" alt="enter image description here"></a></p>
  
  <p>But in case of complex neural network, the input space is much bigger and the sample data points are much more clustered with big chunks of empty space between them:</p>
  
  <p><a href="https://i.sstatic.net/74IAT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/74IAT.png" alt="enter image description here"></a></p>
  
  <p>The neural network then doesn't know how to classify the data in the empty space, so something like this is possible :</p>
  
  <p><a href="https://i.sstatic.net/2VAYc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2VAYc.png" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/Vwep2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Vwep2.png" alt="enter image description here"></a></p>
  
  <p>When using the traditional ways of measuring quality of neural networks, both of these will be considered good. As they do classify the classes themselves correctly.</p>
  
  <p>Then, what happens if we try to classify these data points?</p>
  
  <p><a href="https://i.sstatic.net/Ngw14.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ngw14.png" alt="enter image description here"></a></p>
  
  <p>Really, neural network has no data it could fall back on, so it just outputs what seems to us as random nonsense.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-30T09:00:36.700Z
Answers.LastActivityDate: 2020-01-30T09:00:36.700Z
Answers.UserDisplayName: Euphoric
Answers.Body: <p>In your particular case, you could add a eleventh category to your training data: "not a digit".</p>
  
  <p>Then train your model with a bunch of images of incorrectly segmented digits, in addition to the normal digit examples. This way the model will learn to tell apart real digits from incorrectly segmented ones.</p>
  
  <p>However even after doing that, there will be an infinite number of random looking images that will be classified as digits. They're just far away from the examples of "not a digit" you provided.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-30T13:35:42.510Z
Answers.LastActivityDate: 2020-01-30T13:35:42.510Z
Answers.UserDisplayName: jpa
Answers.Body: <p>The concept you are looking for is called epistemic uncertainty, also known as model uncertainty. You want the model to produce meaningful calibrated probabilities that quantify the real confidence of the model.</p>
  
  <p>This is generally not possible with simple neural networks as they simply do not have this property, for this you need a Bayesian Neural Network (BNN). This kind of network learns a distribution of weights instead of scalar or point-wise weights, which then allow to encode model uncertainty, as then the distribution of the output is calibrated and has the properties you want.</p>
  
  <p>This problem is also called out of distribution (OOD) detection, and again it can be done with BNNs, but unfortunately training a full BNN is untractable, so we use approximations.</p>
  
  <p>As a reference, one of these approximations is Deep Ensembles, which train several instances of a model in the same dataset and then average the softmax probabilities, and has good out of distribution detection properties. Check the paper <a href="https://arxiv.org/abs/1612.01474" rel="noreferrer">here</a>, in particular section 3.5 which shows results for OOD based on entropy of the ensemble probabilities.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T08:50:41.740Z
Answers.Text: BNNs help, but it should be noted that _humans have this problem too_ (and there's a good chance _our_ brains use BNNs). There will _always_ be things that are obviously true for two different systems, even though both "100% certain" (sigh) results are different - the 2015 blue-black/gold-white dress phenomenon should serve as a great example.
Answers.UserDisplayName: Luaan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T09:53:03.207Z
Answers.Text: You can also get an ensemble from a single model if you leave dropout on at inference time: http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html
Answers.UserDisplayName: Davidmh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T09:56:13.733Z
Answers.Text: @Davidmh Actually thats not really an ensemble, and I did not mention it because MC Dropout does not have good out of distribution detection properties, you can see that in the Deep Ensembles paper I linked.
Answers.UserDisplayName: Dr. Snoopy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T16:23:53.740Z
Answers.Text: MC dropout is just an approximation of a BNN (or deep Gaussian process). I think it's worth mentioning it because it can easily be applied (e.g. Keras already provides an implementation of dropout that can be enabled at inference time), so people can easily use it.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-30T05:31:43.757Z
Answers.Text: @AlexanderSoare I don't think that works, because specifying the "other" class is pretty much any class, its not bounded (open set), so you will always have trouble finding enough data to match a Bayesian model (which models the opposite proble, which is bounded).
Answers.UserDisplayName: Dr. Snoopy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-30T12:11:22.140Z
Answers.Text: Decent related paper: [On Calibration of Modern Neural Networks](https://arxiv.org/pdf/1706.04599.pdf)
Answers.UserDisplayName: alkasm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-31T09:47:20.477Z
Answers.Text: I think this is not correct. Since the model has to choose, it is actually correct to say it is 100% more likely to be a 0 than anything else. I don't believe that property changes when using BNN. If the choose would against another valid number, I believe confidence would help.
Answers.UserDisplayName: PascalVKooten
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-31T10:18:13.520Z
Answers.Text: @PascalVKooten I am not sure what you mean by "this is not correct", a BNN effective makes out of distribution predictions with a more flat distribution (which has higher entropy) indicating that it does not know the answer, or even hinting that there are many possible answers. A non-BNN model makes overconfident predictions that are generally wrong.
Answers.UserDisplayName: Dr. Snoopy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-28T16:40:30.243Z
Answers.LastActivityDate: 2020-01-29T06:57:57.360Z
Answers.LastEditDate: 2020-01-29T06:57:57.360Z
Answers.UserDisplayName: Dr. Snoopy
Body: <p>I trained a simple CNN on the MNIST database of handwritten digits to 99% accuracy. I'm feeding in a bunch of handwritten digits, and non-digits from a document.</p>
  
  <p>I want the CNN to report errors, so I set a threshold of 90% certainty below which my algorithm assumes that what it's looking at is not a digit. </p>
  
  <p>My problem is that the CNN is 100% certain of many incorrect guesses. In the example below, the CNN reports 100% certainty that it's a 0. How do I make it report failure?</p>
  
  <p><a href="https://i.sstatic.net/nPBYU.png" rel="noreferrer"><img src="https://i.sstatic.net/nPBYU.png" alt="Handwritten digit input - does not look like a digit"></a></p>
  
  <p><strong>My thoughts on this</strong>:
  Maybe the CNN is not really 100% certain that this is a zero. Maybe it just thinks that it can't be anything else, and it's being forced to choose (because of normalisation on the output vector). Is there any way I can get insight into what the CNN "thought" before I forced it to choose?</p>
  
  <p>PS: I'm using Keras on Tensorflow with Python.</p>
  
  <p><strong>Edit</strong></p>
  
  <p>Because someone asked. Here is the context of my problem:</p>
  
  <p>This came from me applying a heuristic algorithm for segmentation of sequences of connected digits. In the image above, the left part is actually a 4, and the right is the curve bit of a 2 without the base. The algorithm is supposed to step through segment cuts, and when it finds a confident match, remove that cut and continue moving along the sequence. It works really well for some cases, but of course it's totally reliant on being able to tell if what it's looking at is not a good match for a digit. Here's an example of where it kind of did okay.</p>
  
  <p><a href="https://i.sstatic.net/rVDz4.png" rel="noreferrer"><img src="https://i.sstatic.net/rVDz4.png" alt="Various input images for handwritten digit recognition"></a></p>
  
  <p>My next best option is to do inference on all permutations and maximise combined score. That's more expensive.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-28T16:39:59.843Z
Comments.Text: @nbro Just so you know what I've tried. I removed the softmax from the last dense layer so I can see the activations before normalization. It's defintely giving me richer information that I can use for error reporting. I think that maybe if I then apply softmax and use a condition which combines the two metrics, I can improve my error report. In any case, let me know what your answer is when you have time :)
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-28T16:43:59.853Z
Comments.Text: I need to think about your solution and workaround, but I would have given an answer similar to [https://ai.stackexchange.com/a/17722/2444](https://ai.stackexchange.com/a/17722/2444). There's an issue with traditional neural networks, as they are programmed to learn a point estimate so they do not usually model uncertainty.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-29T11:42:53.983Z
Comments.Text: This is very obviously a 2... It's just that the actual digit is a 3D object, and it's being viewed at an unusual angle.
Comments.UserDisplayName: John Dvorak
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-30T04:24:58.750Z
Comments.Text: Going from left to right and also the other way round might help, sometimes. Sometimes, there may be a clearly visible digit on one end but not the other, so some meet-in-the-middle might help. This obviously doesn't solve the problem, but it's not too expensive and might be an idea to try.
Comments.UserDisplayName: maaartinus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-22T17:45:43.840Z
Comments.Text: Can you share your code for this? I'm looking to obtain confidence metrics for my CNN classifier as well as make that subplot that you have that displays all the images with their predicted and actual labels. Greatly appreciate it!
Comments.UserDisplayName: Aziz Uddin
Tags: <convolutional-neural-networks><tensorflow><keras><bayesian-deep-learning><uncertainty-quantification>
Title: Why do CNN's sometimes make highly confident mistakes, and how can one combat this problem?
UserDisplayName: Alexander Soare
ThreadText: # Why do CNN's sometimes make highly confident mistakes, and how can one combat this problem?
  
  (Asked by: username_0 on 2020-01-28)
  
  I trained a simple CNN on the MNIST database of handwritten digits to 99% accuracy. I'm feeding in a bunch of handwritten digits, and non-digits from a document.
  
  I want the CNN to report errors, so I set a threshold of 90% certainty below which my algorithm assumes that what it's looking at is not a digit.
  
  My problem is that the CNN is 100% certain of many incorrect guesses. In the example below, the CNN reports 100% certainty that it's a 0. How do I make it report failure?
  
  [![Handwritten digit input - does not look like a digit](https://i.sstatic.net/nPBYU.png)](https://i.sstatic.net/nPBYU.png)
  
  **My thoughts on this**:
  Maybe the CNN is not really 100% certain that this is a zero. Maybe it just thinks that it can't be anything else, and it's being forced to choose (because of normalisation on the output vector). Is there any way I can get insight into what the CNN "thought" before I forced it to choose?
  
  PS: I'm using Keras on Tensorflow with Python.
  
  **Edit**
  
  Because someone asked. Here is the context of my problem:
  
  This came from me applying a heuristic algorithm for segmentation of sequences of connected digits. In the image above, the left part is actually a 4, and the right is the curve bit of a 2 without the base. The algorithm is supposed to step through segment cuts, and when it finds a confident match, remove that cut and continue moving along the sequence. It works really well for some cases, but of course it's totally reliant on being able to tell if what it's looking at is not a good match for a digit. Here's an example of where it kind of did okay.
  
  [![Various input images for handwritten digit recognition](https://i.sstatic.net/rVDz4.png)](https://i.sstatic.net/rVDz4.png)
  
  My next best option is to do inference on all permutations and maximise combined score. That's more expensive.
  
  **Tags:** convolutional-neural-networks, tensorflow, keras, bayesian-deep-learning, uncertainty-quantification
  > **Comments:**
  > **username_0**: @nbro Just so you know what I've tried. I removed the softmax from the last dense layer so I can see the activations before normalization. It's defintely giving me richer information that I can use for error reporting. I think that maybe if I then apply softmax and use a condition which combines the two metrics, I can improve my error report. In any case, let me know what your answer is when you have time :)  
  > **username_1**: I need to think about your solution and workaround, but I would have given an answer similar to [https://ai.stackexchange.com/a/17722/2444](https://ai.stackexchange.com/a/17722/2444). There's an issue with traditional neural networks, as they are programmed to learn a point estimate so they do not usually model uncertainty.  
  > **username_2**: This is very obviously a 2... It's just that the actual digit is a 3D object, and it's being viewed at an unusual angle.  
  > **username_3**: Going from left to right and also the other way round might help, sometimes. Sometimes, there may be a clearly visible digit on one end but not the other, so some meet-in-the-middle might help. This obviously doesn't solve the problem, but it's not too expensive and might be an idea to try.  
  > **username_4**: Can you share your code for this? I'm looking to obtain confidence metrics for my CNN classifier as well as make that subplot that you have that displays all the images with their predicted and actual labels. Greatly appreciate it!  
  
  ---
  
  ## 6 answers
  
  ### Answer by username_13 on 2020-01-28. Score: 35 | Accepted answer
  
  The concept you are looking for is called epistemic uncertainty, also known as model uncertainty. You want the model to produce meaningful calibrated probabilities that quantify the real confidence of the model.
  
  This is generally not possible with simple neural networks as they simply do not have this property, for this you need a Bayesian Neural Network (BNN). This kind of network learns a distribution of weights instead of scalar or point-wise weights, which then allow to encode model uncertainty, as then the distribution of the output is calibrated and has the properties you want.
  
  This problem is also called out of distribution (OOD) detection, and again it can be done with BNNs, but unfortunately training a full BNN is untractable, so we use approximations.
  
  As a reference, one of these approximations is Deep Ensembles, which train several instances of a model in the same dataset and then average the softmax probabilities, and has good out of distribution detection properties. Check the paper [here](https://arxiv.org/abs/1612.01474), in particular section 3.5 which shows results for OOD based on entropy of the ensemble probabilities.
  
  > **Comments:**
  > **username_14**: BNNs help, but it should be noted that _humans have this problem too_ (and there's a good chance _our_ brains use BNNs). There will _always_ be things that are obviously true for two different systems, even though both "100% certain" (sigh) results are different - the 2015 blue-black/gold-white dress phenomenon should serve as a great example.  
  > **username_15**: You can also get an ensemble from a single model if you leave dropout on at inference time: http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html  
  > **username_13**: @Davidmh Actually thats not really an ensemble, and I did not mention it because MC Dropout does not have good out of distribution detection properties, you can see that in the Deep Ensembles paper I linked.  
  > **username_1**: MC dropout is just an approximation of a BNN (or deep Gaussian process). I think it's worth mentioning it because it can easily be applied (e.g. Keras already provides an implementation of dropout that can be enabled at inference time), so people can easily use it.  
  > **username_13**: @AlexanderSoare I don't think that works, because specifying the "other" class is pretty much any class, its not bounded (open set), so you will always have trouble finding enough data to match a Bayesian model (which models the opposite proble, which is bounded).  
  > **username_16**: Decent related paper: [On Calibration of Modern Neural Networks](https://arxiv.org/pdf/1706.04599.pdf)  
  > **username_17**: I think this is not correct. Since the model has to choose, it is actually correct to say it is 100% more likely to be a 0 than anything else. I don't believe that property changes when using BNN. If the choose would against another valid number, I believe confidence would help.  
  > **username_13**: @PascalVKooten I am not sure what you mean by "this is not correct", a BNN effective makes out of distribution predictions with a more flat distribution (which has higher entropy) indicating that it does not know the answer, or even hinting that there are many possible answers. A non-BNN model makes overconfident predictions that are generally wrong.  
  
  ---
  
  ### Answer by username_5 on 2020-01-29. Score: 15
  
  Your classifier is specifically learning the ways in which **0**s are different from other digits,
  **not** what it really means for a digit to **be** a zero.
  
  Philosophically, you could say the model appears to have some powerful understanding when restricted
  to a tightly controlled domain, but that facade is lifted as soon as you throw any sort of wrench in
  the works.
  
  Mathematically, you could say that the model is simply optimizing a classification metric for data
  drawn from a specific distribution, and when you give it data from a different distribution, all
  bets are off.
  
  The go-to answer is to collect or generate data like the data you expect the model to deal with
  (in practice, the effort required to do so can vary dramatically depending upon the application). In
  this case, that could involve drawing a bunch of random scribbles and adding them to your training
  data set. At this point you must ask, now how do I label them? You will want a new "other" or
  "non-digit" class so that your model can learn to categorize these scribbles separately from digits.
  After retraining, your model should now better deal with these cases.
  
  However, you may then ask, but what if I gave it color images of digits? Or color images of farm
  animals? Maybe pigs will be classified as zeros because they are round. This problem is a
  fundamental property of the way deep learning is orchestrated. Your model is not capable of higher
  order logic, which means it can seem to go from being very intelligent to very dumb by just throwing
  the slightest curve ball at it. For now, all deep learning does is recognize patterns in data that
  allow it to minimize some loss function.
  
  Deep learning is a fantastic tool, but not an all-powerful omnitool. Bear in mind its limitations
  and use it where appropriate, and it will serve you well.
  
  > **Comments:**
  > **username_0**: @Fnguyen yes totally agree there. Why wouldn't it at least think that maybe it's looking at something else. Although I'm not really sure how overfitting would cause this issue. Unless there were many zeros in the training set that looked like this image. For reference I used MNIST, then post trained on a set of 3000 digits with slightly different handwriting styles (turns out South Koreans write digits a bit different than westeners).  
  
  ---
  
  ### Answer by username_10 on 2020-01-29. Score: 7
  
  ## Broken assumptions
  
  Generalization relies on making strong assumptions (no free lunch, etc). If you break your assumptions, then you're not going to have a good time. A key assumption of a standard digit-recognition classifier like MNIST is that you're classifying pictures that actually contain a single digit. If your real data contains pictures that have non-digits, then that means that your real data is *not* similar to training data but is conceptually very, very different.
  
  If that's a problem (as in this case) then one way to treat that is to explicitly break that assumption and train a model that not only recognizes digits 0-9 but also recognizes whether there's a digit at all, and is able to provide an answer "that's not a digit", so a 11-class classifier instead of a 10-class one. MNIST training data is not sufficient for that, but you can use some kind of 'distractor' data to provide the not-a-digit examples. For example, you could use some dataset of letters (perhaps omitting I, l, O and B) transformed to look similar to MNIST data.
  
  
  ---
  
  ### Answer by username_6 on 2020-01-29. Score: 3
  
  Apollys,
  
  That's a very well thought out response. Particularly, the philosophical discussion of the essence of "0-ness."
  
  I haven't actually performed this experiment, so caveat emptor... I wonder how well an "other" class would actually work. The ways in which "other" differs from "digit" has infinite variability (or at least its only limitation is the cardinality of the input layer).
  
  The NN decides whether something is more of one class or more of a different class. If there isn't an essence in common among other "non-digits", I don't believe it will do well at identifying "other" as the catch-all for everything that has low confidence level of classification.
  
  This approach still doesn't identify what it is to ***be*** "not-digit". It identifies how all the things that are "other" differ from the other labeled inputs -- probably poorly, depending on the variability of the "non-digit" labeled data. (i.e. is it numerically exhaustive, many times over, of all random scribbles?) Thoughts?
  
  > **Comments:**
  > **username_7**: This answer looks like it should be a comment, but it looks like you don't have enough reputation to comment. You brought up some great concerns about the effectiveness of adding an "other" class. It can be really hard to predict what will or won't work well with neural networks.  
  > **username_0**: Totally agree here. I was thinking something along the lines when I saw the suggestion. @MatiasValdenegro (don't know if tagging you here works), I was wondering what your thoughts on such an approach might be.  
  > **username_5**: Yes, this is an important point.  Once again, the "other" class will have to come from some specific distribution and have enough samples to cover that distribution.  That means if you really want your model to learn what it means to be a digit rather than anything else that is possible, you will need a (probably intractably) **lot** of data.  In this way, we see that deep learning is largely about understanding and confining the problem domain very well.  
  > **username_8**: Point from practical experience: having _one_ other class may work, but in some cases it can help to have multiple. I'm not aware of a method to determine up front what the best number of "other" classes is.  
  > **username_9**: An approach may be to use/train some form of binary classifier to discard non-digit examples prior to the recognition step (i.e., 0-9 classification). You'll have to maximise recall of the digit inputs, so there may still be a few non-digits slip through, but such an approach may reduce the detriment to the performance of classifying the real digits.  
  
  ---
  
  ### Answer by username_11 on 2020-01-30. Score: 2
  
  I'm an amateur with neural networks, but I will illustrate my understanding of how this problem comes to be.
  
  First, lets see how trivial neural network classifies 2D input into two classes :
  [![enter image description here](https://i.sstatic.net/OgldC.png)](https://i.sstatic.net/OgldC.png)
  
  But in case of complex neural network, the input space is much bigger and the sample data points are much more clustered with big chunks of empty space between them:
  
  [![enter image description here](https://i.sstatic.net/74IAT.png)](https://i.sstatic.net/74IAT.png)
  
  The neural network then doesn't know how to classify the data in the empty space, so something like this is possible :
  
  [![enter image description here](https://i.sstatic.net/2VAYc.png)](https://i.sstatic.net/2VAYc.png)
  [![enter image description here](https://i.sstatic.net/Vwep2.png)](https://i.sstatic.net/Vwep2.png)
  
  When using the traditional ways of measuring quality of neural networks, both of these will be considered good. As they do classify the classes themselves correctly.
  
  Then, what happens if we try to classify these data points?
  
  [![enter image description here](https://i.sstatic.net/Ngw14.png)](https://i.sstatic.net/Ngw14.png)
  
  Really, neural network has no data it could fall back on, so it just outputs what seems to us as random nonsense.
  
  
  ---
  
  ### Answer by username_12 on 2020-01-30. Score: 0
  
  In your particular case, you could add a eleventh category to your training data: "not a digit".
  
  Then train your model with a bunch of images of incorrectly segmented digits, in addition to the normal digit examples. This way the model will learn to tell apart real digits from incorrectly segmented ones.
  
  However even after doing that, there will be an infinite number of random looking images that will be classified as digits. They're just far away from the examples of "not a digit" you provided.
  
  
  ---
  
  