Answers.Body: <p>Your question seems to be talking about two slightly different topics:</p>
  <ul>
  <li>Pros and cons of 'one vs rest' approach in multi-class classification</li>
  <li>Use of Neural Networks in single-output vs multi-class classification problems</li>
  </ul>
  <p><strong>One vs Rest in Multi-Class Classification</strong></p>
  <p>Recognising digits is an example of multi-class classification. The approach you outline is the kind of approach summarised in the &quot;One vs Rest&quot; section of the <a href="https://en.wikipedia.org/wiki/Multiclass_classification" rel="nofollow noreferrer">Wikipedia page on multi-class classification</a>. The page notes the following issues with this approach:</p>
  <blockquote>
  <p>Firstly, the scale of the confidence values may differ between the binary classifiers. Second, even if the class distribution is balanced in the training set, the binary classification learners see unbalanced distributions because typically the set of negatives they see is much larger than the set of positives.</p>
  </blockquote>
  <p>You might also like to look into another approach called One vs One (<a href="https://machinelearningmastery.com/one-vs-rest-and-one-vs-one-for-multi-class-classification/" rel="nofollow noreferrer">'One vs Rest' vs 'One vs One'</a>) which sets up the classification problem as a set of binary alternatives. In the digit recognition case you'd end up with a classifier for &quot;1 or 2?&quot;, &quot;1 or 3?&quot;, &quot;1 or 4?&quot; etc. This might help with the &quot;4 vs 9&quot; problem but it does mean an enormous amount of classifiers, that might be better represented in some kind of network. Perhaps even a network inspired by brain neurons.</p>
  <p><strong>Use of Neural Networks in single output vs multi-class classification</strong></p>
  <p>There is nothing magical about a neural network that means it has to be used for multi-class classification. Nor is there anything magical about it that makes it the only option for multi-class classification.</p>
  <p>For example:</p>
  <ul>
  <li><a href="https://builtin.com/data-science/how-build-neural-network-keras" rel="nofollow noreferrer">Using a Neural Network for sentiment analysis</a> outputs one single answer about how positive/negative a piece of text is.</li>
  <li><a href="https://github.com/pramodini18/Digit-recognition-using-SVM" rel="nofollow noreferrer">Digit recognition using SVM</a> uses something that isn't a neural network for multi-class classification</li>
  </ul>
  <p><strong>Conclusions</strong></p>
  <p>A 10-class neural network is used to identify digits because this has turned out to be an efficient way of doing so when compared with one vs rest and one vs all approaches.</p>
  <p>A bit off-topic, perhaps, but if you think about this in the context of <a href="https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html" rel="nofollow noreferrer">T5</a>, there does seem to be a trend of moving towards larger more multi-purpose models rather than lots of small specialised models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-02T09:23:13.183Z
Answers.LastActivityDate: 2021-01-02T09:23:13.183Z
Answers.UserDisplayName: Alan Buxton
Answers.Body: <p>In practice you never want to classify just a single digit rather than series. In such case you have to pass a patch of image to multiple network, which would make it inconvenient. If you built different accurate models, training parameters will not significantly reduced. For example sloppy written 6, in a single model the probability of being 6 and 0 would be close, not same if you consider likelihood you may get closest answer. While with different models the probability may vary in a greater scale and you may not have good generalization as you have in single model. At the end everything boils down to generalization and in my experience neural networks trained with multiple things have good generalization property that a single.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-01T14:05:39.720Z
Answers.LastActivityDate: 2020-10-01T14:05:39.720Z
Answers.UserDisplayName: Darshan
Answers.Body: <p>Imagine a small kid who has no idea about the world around it. You teach the kid how to write the number &quot;6&quot; and that is the only thing that it knows.</p>
  <p>Now, No matter what other number you show the kid , it's gonna always respond with &quot;6&quot; because that is the only thing it knows or it has learned.</p>
  <p>You teach the kid how to write the number &quot;9&quot;, so now it knows how to differentiate a &quot;6&quot; from a &quot;9&quot; and no matter what other number you show the kid, there is a 50 % chance of it responding with a &quot;6&quot; or a &quot;9&quot; because it knows only that much.</p>
  <p>The purpose of a neural network is to understand underlying distribution in data that can help it in classifying different numbers. It's important to have a classifier that understands general characteristics of numbers and help us with our task.
  If you have 10 neural networks trained on 10 different digits, and you show each of these networks the number &quot;10&quot;, each network will output the number on which it was trained because that is all it knows(similar to the naive kid above).</p>
  <p>I hope this answers your question!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-03T02:07:11.420Z
Answers.Text: I guess the kid would know "6 or not 6" in this case? So in case of 10 different neural networks, only one of them would answer "yes 10" on well trained networks.
Answers.UserDisplayName: Ville
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-03T02:11:06.947Z
Answers.Text: Your question was on why does a neural network need to be trained on multiple labels instead of one right? In "6" vs "not 6", the model is already trained on 2 labels
Answers.UserDisplayName: Srivatsan Ramesh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-03T02:27:06.313Z
Answers.Text: Nope, there would be a confidence between 0 and 1 wether it is the number 6 or not. Only 1 label needed.
Answers.UserDisplayName: Ville
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-03T04:08:05.847Z
Answers.Text: I think the question should have been rephrased to recognising multiple digits over labels because labels are local to a model, for a single model to identify whether it is 10 or not, it still needs 2 labels(10 or not 10) whereas it is recognising one digit. Anyways to answer your question on having 10 models recognise 10 digits vs 1 model recognising 10 digits, it's just convenience! Would you have 10 people recognise 10 digits for you or 1 person with the ability to do the work of 10 people? There is nothing wrong with either! It's just convenience I guess!
Answers.UserDisplayName: Srivatsan Ramesh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-03T01:52:28.477Z
Answers.LastActivityDate: 2020-12-03T01:52:28.477Z
Answers.UserDisplayName: Srivatsan Ramesh
Body: <p>I was watching this series: <a href="https://www.youtube.com/watch?v=aircAruvnKk" rel="nofollow noreferrer">https://www.youtube.com/watch?v=aircAruvnKk</a></p>
  <p>The series demonstrates neural networks by building a simple number recognizing network.</p>
  <p>It got me thinking: Why neural networks try to recognize multiple labels instead of just one? In the above example, the network tries to recognize numbers from 0 to 9. What is the benefit of trying to recognize so many things simultaneously? Wouldn't it make it easier to reason about if there would be 10 different neural networks which would specialize to recognize only one number at a time?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-30T15:17:12.093Z
Comments.Text: You could train 10 different models in this case. Model one would then be trained to learn is this a zero or not a zero, and so forth. This would increase the number of negative training samples for each model. But what if then there comes a case where you want to use this in practice? A: you would have to perform 10 forward passes instead of just one. B: the case were the input is a sloppy written 4 that looks similar to a 9, if you have one model then it could learn those small slight variations, if you have 10 different, the difference between a 4 and a 9 would not be learned.
Comments.UserDisplayName: Isbister
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-01T12:51:22.457Z
Comments.Text: Thank you for the answer! But wouldn't single purpose networks make each individual network simpler? So you could have less neurons and so on. Which (I believe) would make them more performant. Is it 10x more performant than a bigger network (so that it would even out running the picture trough 10 trained networks)? dunno.
Comments.UserDisplayName: Ville
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-01T12:51:42.990Z
Comments.Text: The point about difference between 4 and 9 kind of cases is good, but on the other hand, couldn't both neurons still result similar probability for 9s and 4s even if they would live in the same network? And when you would train e.g number 4, you would still feed 9's to it for negative results, so the learning would still happen I believe. Would it be as well learnt? Not sure and would be nice to understand which way it is. I'm trying to understand the practical differences and pros and cons of these approaches.
Comments.UserDisplayName: Ville
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-31T14:56:02.660Z
Comments.Text: Multiple specialized networks are called Ensemble Networks. It may indeed be worth training it that way if you are teaching a CNN various representations of the same number. It'd just be computationally more expensive to train and run. If you want to compare, then create programs as a single and ensemble network and try. Learn by doing.
Comments.UserDisplayName: Nav
Tags: <neural-networks>
Title: Why neural networks tend to be trained to recognize multiple things instead of just one?
UserDisplayName: Ville
ThreadText: # Why neural networks tend to be trained to recognize multiple things instead of just one?
  
  (Asked by: username_0 on 2020-09-30)
  
  I was watching this series: <https://www.youtube.com/watch?v=aircAruvnKk>
  
  The series demonstrates neural networks by building a simple number recognizing network.
  
  It got me thinking: Why neural networks try to recognize multiple labels instead of just one? In the above example, the network tries to recognize numbers from 0 to 9. What is the benefit of trying to recognize so many things simultaneously? Wouldn't it make it easier to reason about if there would be 10 different neural networks which would specialize to recognize only one number at a time?
  
  **Tags:** neural-networks
  > **Comments:**
  > **username_1**: You could train 10 different models in this case. Model one would then be trained to learn is this a zero or not a zero, and so forth. This would increase the number of negative training samples for each model. But what if then there comes a case where you want to use this in practice? A: you would have to perform 10 forward passes instead of just one. B: the case were the input is a sloppy written 4 that looks similar to a 9, if you have one model then it could learn those small slight variations, if you have 10 different, the difference between a 4 and a 9 would not be learned.  
  > **username_0**: Thank you for the answer! But wouldn't single purpose networks make each individual network simpler? So you could have less neurons and so on. Which (I believe) would make them more performant. Is it 10x more performant than a bigger network (so that it would even out running the picture trough 10 trained networks)? dunno.  
  > **username_0**: The point about difference between 4 and 9 kind of cases is good, but on the other hand, couldn't both neurons still result similar probability for 9s and 4s even if they would live in the same network? And when you would train e.g number 4, you would still feed 9's to it for negative results, so the learning would still happen I believe. Would it be as well learnt? Not sure and would be nice to understand which way it is. I'm trying to understand the practical differences and pros and cons of these approaches.  
  > **username_2**: Multiple specialized networks are called Ensemble Networks. It may indeed be worth training it that way if you are teaching a CNN various representations of the same number. It'd just be computationally more expensive to train and run. If you want to compare, then create programs as a single and ensemble network and try. Learn by doing.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2021-01-02. Score: 1
  
  Your question seems to be talking about two slightly different topics:
  
  * Pros and cons of 'one vs rest' approach in multi-class classification
  * Use of Neural Networks in single-output vs multi-class classification problems
  
  **One vs Rest in Multi-Class Classification**
  
  Recognising digits is an example of multi-class classification. The approach you outline is the kind of approach summarised in the "One vs Rest" section of the [Wikipedia page on multi-class classification](https://en.wikipedia.org/wiki/Multiclass_classification). The page notes the following issues with this approach:
  
  > Firstly, the scale of the confidence values may differ between the binary classifiers. Second, even if the class distribution is balanced in the training set, the binary classification learners see unbalanced distributions because typically the set of negatives they see is much larger than the set of positives.
  
  You might also like to look into another approach called One vs One (['One vs Rest' vs 'One vs One'](https://machinelearningmastery.com/one-vs-rest-and-one-vs-one-for-multi-class-classification/)) which sets up the classification problem as a set of binary alternatives. In the digit recognition case you'd end up with a classifier for "1 or 2?", "1 or 3?", "1 or 4?" etc. This might help with the "4 vs 9" problem but it does mean an enormous amount of classifiers, that might be better represented in some kind of network. Perhaps even a network inspired by brain neurons.
  
  **Use of Neural Networks in single output vs multi-class classification**
  
  There is nothing magical about a neural network that means it has to be used for multi-class classification. Nor is there anything magical about it that makes it the only option for multi-class classification.
  
  For example:
  
  * [Using a Neural Network for sentiment analysis](https://builtin.com/data-science/how-build-neural-network-keras) outputs one single answer about how positive/negative a piece of text is.
  * [Digit recognition using SVM](https://github.com/pramodini18/Digit-recognition-using-SVM) uses something that isn't a neural network for multi-class classification
  
  **Conclusions**
  
  A 10-class neural network is used to identify digits because this has turned out to be an efficient way of doing so when compared with one vs rest and one vs all approaches.
  
  A bit off-topic, perhaps, but if you think about this in the context of [T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html), there does seem to be a trend of moving towards larger more multi-purpose models rather than lots of small specialised models.
  
  
  ---
  
  ### Answer by username_4 on 2020-10-01. Score: 1
  
  In practice you never want to classify just a single digit rather than series. In such case you have to pass a patch of image to multiple network, which would make it inconvenient. If you built different accurate models, training parameters will not significantly reduced. For example sloppy written 6, in a single model the probability of being 6 and 0 would be close, not same if you consider likelihood you may get closest answer. While with different models the probability may vary in a greater scale and you may not have good generalization as you have in single model. At the end everything boils down to generalization and in my experience neural networks trained with multiple things have good generalization property that a single.
  
  
  ---
  
  ### Answer by username_5 on 2020-12-03. Score: 0
  
  Imagine a small kid who has no idea about the world around it. You teach the kid how to write the number "6" and that is the only thing that it knows.
  
  Now, No matter what other number you show the kid , it's gonna always respond with "6" because that is the only thing it knows or it has learned.
  
  You teach the kid how to write the number "9", so now it knows how to differentiate a "6" from a "9" and no matter what other number you show the kid, there is a 50 % chance of it responding with a "6" or a "9" because it knows only that much.
  
  The purpose of a neural network is to understand underlying distribution in data that can help it in classifying different numbers. It's important to have a classifier that understands general characteristics of numbers and help us with our task.
  If you have 10 neural networks trained on 10 different digits, and you show each of these networks the number "10", each network will output the number on which it was trained because that is all it knows(similar to the naive kid above).
  
  I hope this answers your question!
  
  > **Comments:**
  > **username_0**: I guess the kid would know "6 or not 6" in this case? So in case of 10 different neural networks, only one of them would answer "yes 10" on well trained networks.  
  > **username_5**: Your question was on why does a neural network need to be trained on multiple labels instead of one right? In "6" vs "not 6", the model is already trained on 2 labels  
  > **username_0**: Nope, there would be a confidence between 0 and 1 wether it is the number 6 or not. Only 1 label needed.  
  > **username_5**: I think the question should have been rephrased to recognising multiple digits over labels because labels are local to a model, for a single model to identify whether it is 10 or not, it still needs 2 labels(10 or not 10) whereas it is recognising one digit. Anyways to answer your question on having 10 models recognise 10 digits vs 1 model recognising 10 digits, it's just convenience! Would you have 10 people recognise 10 digits for you or 1 person with the ability to do the work of 10 people? There is nothing wrong with either! It's just convenience I guess!  
  
  ---
  
  