Answers.Body: <p>The most obvious way more classes increase the network size is the output layer, but I don't believe there is a rule of thumb for the size of the entire network.</p>
  <p>As I understand it, there is no clear answer to how big a network needs to be to achieve a certain performance with regard to the number of layers compared to the number of classes. This is a very active research field, just as an example compare the size of EfficientNet to other State of the art models when it was introduced and you can see the size difference.</p>
  <p>Regarding the data needed, in <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer">The Deep Learning Book</a> (which is a few years old now) they state as a general that with the models available at that time, you needed ~5000 examples per label for acceptable performance, while to exceed human performance (their words) you would need about 10 million labeled examples.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-25T13:35:48.810Z
Answers.Text: I didn't expect the difference between acceptable and super-human to be three orders of magnitude. Do you remember in which chapter of the book this statement was made?
Answers.UserDisplayName: conscious_process
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-25T17:31:35.373Z
Answers.Text: As I mentioned, it is a rather old book (with regard to DL, since everything develops so fast) so this might have changed since it's a rather active research field. However, I think it's in the introductory chapter, under some title with Data in the name.
Answers.UserDisplayName: Nathanson
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-25T10:24:36.970Z
Answers.LastActivityDate: 2021-12-21T04:06:07.240Z
Answers.LastEditDate: 2021-12-21T04:06:07.240Z
Answers.UserDisplayName: Nathanson
Answers.Body: <p>Model/network design has multiple guidelines, a basic one is: The solving capacity of the network should be larger than the possibility space of the problem to be solved.</p>
  <p>Solving capacity (<strong>learning capacity</strong>) of a network (dense usually) can be calculated as the product of number of neurons in all layers, for example:</p>
  <p>Input shape: 10 values <br/>
  Network shape: [layer1 30 units, layer2 20 units, output 1 unit] should have learning capacity of <span class="math-container">$30 \times 20 \times 1 = 600$</span>, it learn roughly max 600 different inputs (each input holds 10 values).</p>
  <p>Another consideration, <strong>the separation lines</strong>, even when the inputs of the problem to be learnt are unlimited, but the 2 classes (just example) are always separated on 2 sides of a line without mixing up, just a single neuron can solve the problem.</p>
  <p>One neuron can make 1 separation line, 1 layer makes a poly-line with segments are by the neurons in that layer, another layer makes another poly-line.</p>
  <p>Thus, more classes, more separations to be done, and more classes would mean the input variety is large, so surely training data are a lot and model size needs to be large.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-26T08:33:36.003Z
Answers.Text: Very good point with the seperation lines. I am thinking about testing it out by making a dataset generator for n-classes (n-arm spiral?) and see how the size of minimal viable model changes when n increases. 
  
  Imo the number of different inputs a network "can hold" depends on the complexity of the problem. If dataset is seperatable by a line, then it can hold infinite amount of points, because it generalized well. If dataset is completely random, then the network can't generalize and points can only be memorized. Maybe in the second case your product-based method holds, but idk.
Answers.UserDisplayName: conscious_process
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-26T13:44:54.590Z
Answers.Text: yeah, those 'completely random' datasets are crazy
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-25T10:27:06.763Z
Answers.LastActivityDate: 2021-12-21T04:52:14.273Z
Answers.LastEditDate: 2021-12-21T04:52:14.273Z
Answers.UserDisplayName: Dan D.
Body: <p>When solving a classification problem with neural nets, be it text or images, how does the number of classes affect the model size and amount of data needed to train?</p>
  <p>Are there any soft or hard limitations where the number of outputs starts to stall learning?</p>
  <p>Do you know about any analysis of how the number of classes scales the model?</p>
  <p>Does the optimal size increase proportionally with the number of outputs? Does it increase at all? If it does increase, is the relationship linear or exponential?</p>
  
Tags: <neural-networks><classification><computational-learning-theory><architecture><capacity>
Title: In classification, how does the number of classes affect the model size and amount of data needed to train?
UserDisplayName: conscious_process
ThreadText: # In classification, how does the number of classes affect the model size and amount of data needed to train?
  
  (Asked by: username_0 on 2021-02-25)
  
  When solving a classification problem with neural nets, be it text or images, how does the number of classes affect the model size and amount of data needed to train?
  
  Are there any soft or hard limitations where the number of outputs starts to stall learning?
  
  Do you know about any analysis of how the number of classes scales the model?
  
  Does the optimal size increase proportionally with the number of outputs? Does it increase at all? If it does increase, is the relationship linear or exponential?
  
  **Tags:** neural-networks, classification, computational-learning-theory, architecture, capacity
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-02-25. Score: 1
  
  The most obvious way more classes increase the network size is the output layer, but I don't believe there is a rule of thumb for the size of the entire network.
  
  As I understand it, there is no clear answer to how big a network needs to be to achieve a certain performance with regard to the number of layers compared to the number of classes. This is a very active research field, just as an example compare the size of EfficientNet to other State of the art models when it was introduced and you can see the size difference.
  
  Regarding the data needed, in [The Deep Learning Book](https://www.deeplearningbook.org/) (which is a few years old now) they state as a general that with the models available at that time, you needed ~5000 examples per label for acceptable performance, while to exceed human performance (their words) you would need about 10 million labeled examples.
  
  > **Comments:**
  > **username_0**: I didn't expect the difference between acceptable and super-human to be three orders of magnitude. Do you remember in which chapter of the book this statement was made?  
  > **username_1**: As I mentioned, it is a rather old book (with regard to DL, since everything develops so fast) so this might have changed since it's a rather active research field. However, I think it's in the introductory chapter, under some title with Data in the name.  
  
  ---
  
  ### Answer by username_2 on 2021-02-25. Score: 1
  
  Model/network design has multiple guidelines, a basic one is: The solving capacity of the network should be larger than the possibility space of the problem to be solved.
  
  Solving capacity (**learning capacity**) of a network (dense usually) can be calculated as the product of number of neurons in all layers, for example:
  
  Input shape: 10 values   
  Network shape: [layer1 30 units, layer2 20 units, output 1 unit] should have learning capacity of $30 \times 20 \times 1 = 600$, it learn roughly max 600 different inputs (each input holds 10 values).
  
  Another consideration, **the separation lines**, even when the inputs of the problem to be learnt are unlimited, but the 2 classes (just example) are always separated on 2 sides of a line without mixing up, just a single neuron can solve the problem.
  
  One neuron can make 1 separation line, 1 layer makes a poly-line with segments are by the neurons in that layer, another layer makes another poly-line.
  
  Thus, more classes, more separations to be done, and more classes would mean the input variety is large, so surely training data are a lot and model size needs to be large.
  
  > **Comments:**
  > **username_0**: Very good point with the seperation lines. I am thinking about testing it out by making a dataset generator for n-classes (n-arm spiral?) and see how the size of minimal viable model changes when n increases.   Imo the number of different inputs a network "can hold" depends on the complexity of the problem. If dataset is seperatable by a line, then it can hold infinite amount of points, because it generalized well. If dataset is completely random, then the network can't generalize and points can only be memorized. Maybe in the second case your product-based method holds, but idk.  
  > **username_2**: yeah, those 'completely random' datasets are crazy  
  
  ---
  
  