Answers.Body: <p>This may sound counter intuitive but one of the biggest rules of thumb for model capacity in deep learning:</p>
  
  <p><strong>IT SHOULD OVERFIT</strong>. </p>
  
  <p>Once you get a model to overfit, its easier to experiment with regularizations, module replacements, etc. But in general, it gives you a good starting ground.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T04:08:24.047Z
Answers.Text: Although I understand your point, this sounds like a bad suggestion for several reasons. Ideally, you want to design the neural network before training it and not design by trial-and-error (which can be computationally prohibitive, if your task is difficult, you have a big dataset, big net, etc). What if the training data isn't really similar to the test data, even though this assumption is usually implicitly made? This rule of thumb could have been just a comment.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T04:09:59.853Z
Answers.Text: Maybe you can rephrase this answer by saying that "In theory, if the model overfits the training data, it means that it has a sufficient capacity to model the training data, but this will require you to first train the model and try different configurations".
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T11:18:17.057Z
Answers.Text: Thanks for the tip. And @nbro thanks for the clarity.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T03:47:05.843Z
Answers.LastActivityDate: 2020-02-29T03:47:05.843Z
Answers.UserDisplayName: mshlis
Answers.Body: <p>Personally, when I begin designing a machine learning model, I consider the following points:</p>
  <ul>
  <li><p>My data: if I have simple images, like MNIST ones, or in general images with very low resolution, a very deep network is <strong>not</strong> required.</p>
  </li>
  <li><p>If my problem statement needs to learn a lot of features from each image, such as for the human face, I may need to learn eyes, nose, lips, expressions through their combinations, then I <strong>need</strong> a deep network with convolutional layers.</p>
  </li>
  <li><p>If I have time-series data, LSTM or GRU makes sense, but, I also consider recurrent setup when my data has high resolution, low count data points.</p>
  </li>
  </ul>
  <p>The upper limit however may get decided by resources available on the computing device you are using for training.</p>
  <p>Hope this helps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-28T03:45:52.887Z
Answers.LastActivityDate: 2020-11-13T20:41:24.067Z
Answers.LastEditDate: 2020-11-13T20:41:24.067Z
Answers.UserDisplayName: anurag
Answers.Body: <h2>Theoretical results</h2>
  <p>Rather than providing a rule of thumb (which can be misleading, so I am not a big fan of them), I will provide some theoretical results (the first one is also reported in paper <a href="http://dstath.users.uth.gr/papers/IJRS2009_Stathakis.pdf" rel="nofollow noreferrer">How many hidden layers and nodes?</a>), from which you may be able to derive your rules of thumb, depending on your problem, etc.</p>
  <h3>Result 1</h3>
  <p>The paper <a href="https://pdfs.semanticscholar.org/064f/1e85984b207c1eb3c53ac8b68037089b7a0b.pdf" rel="nofollow noreferrer">Learning capability and storage capacity of two-hidden-layer feedforward networks</a> proves that a 2-hidden layer feedforward
  network (<span class="math-container">$F$</span>) with <span class="math-container">$$2 \sqrt{(m + 2)N} \ll N$$</span> hidden neurons can learn any <span class="math-container">$N$</span> distinct samples <span class="math-container">$D= \{ (x_i, t_i) \}_{i=1}^N$</span> with an arbitrarily small error, where <span class="math-container">$m$</span> is the required number of output neurons. Conversely, a <span class="math-container">$F$</span> with <span class="math-container">$Q$</span> hidden neurons can store at least <span class="math-container">$\frac{Q^2}{4(m+2)}$</span> any distinct data <span class="math-container">$(x_i, t_i)$</span> with
  any desired precision.</p>
  <p>They suggest that a sufficient number of neurons in the first layer should be <span class="math-container">$\sqrt{(m + 2)N} + 2\sqrt{\frac{N}{m + 2}}$</span> and in the second layer should be <span class="math-container">$m\sqrt{\frac{N}{m + 2}}$</span>. So, for example, if your dataset has size <span class="math-container">$N=10$</span> and you have <span class="math-container">$m=2$</span> output neurons, then you should have the first hidden layer with roughly 10 neurons and the second layer with roughly 4 neurons. (I haven't actually tried this!)</p>
  <p>However, these bounds are suited for fitting the training data (i.e. for overfitting), which isn't usually the goal, i.e. you want the network to generalize to unseen data.</p>
  <p>This result is strictly related to the universal approximation theorems, i.e. a network with a single hidden layer can, in theory, approximate any continuous function.</p>
  <h2>Model selection, complexity control, and regularisation</h2>
  <p>There are also the concepts of <em>model selection</em> and <em>complexity control</em>, and there are multiple related techniques that take into account the complexity of the model. The paper <a href="https://pdfs.semanticscholar.org/064f/1e85984b207c1eb3c53ac8b68037089b7a0b.pdf" rel="nofollow noreferrer">Model complexity control and statistical learning theory</a> (2002) may be useful. It is also important to note regularisation techniques can be thought of as controlling the complexity of the model [<a href="https://www.nature.com/articles/s41467-020-14663-9" rel="nofollow noreferrer">1</a>].</p>
  <h2>Further reading</h2>
  <p>You may also want to take a look at these related questions</p>
  <ul>
  <li><p><a href="https://stats.stackexchange.com/q/181/82135">How to choose the number of hidden layers and nodes in a feedforward neural network?</a></p>
  </li>
  <li><p><a href="https://ai.stackexchange.com/q/17870/2444">How to estimate the capacity of a neural network?</a></p>
  </li>
  </ul>
  <p>(I will be updating this answer, as I find more theoretical results or other useful info)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-05T06:46:54.817Z
Answers.LastActivityDate: 2020-11-13T20:40:01.093Z
Answers.LastEditDate: 2020-11-13T20:40:01.093Z
Answers.UserDisplayName: nbro
Body: <p>To give an example. Let's just consider the MNIST dataset of handwritten digits. Here are some things which might have an impact on the optimum model capacity:</p>
  <ul>
  <li>There are 10 output classes</li>
  <li>The inputs are 28x28 grayscale pixels (I think this indirectly affects the model capacity. eg: if the inputs were 5x5 pixels, there wouldn't be much room for varying the way an 8 looks)</li>
  </ul>
  <p><strong>So, is there any way of knowing what the model capacity ought to be?</strong> Even if it's not exact? Even if it's a qualitative understanding of the type &quot;if X goes up, then Y goes down&quot;?</p>
  <p>Just to accentuate what I mean when I say &quot;not exact&quot;: I can already tell that a 100 variable model won't solve MNIST, so at least I have a lower bound. I'm also pretty sure that a 1,000,000,000 variable model is way more than needed. Of course, knowing a smaller range than that would be much more useful!</p>
  
Tags: <neural-networks><computational-learning-theory><regularization><vc-dimension><capacity>
Title: Are there any rules of thumb for having some idea of what capacity a neural network needs to have for a given problem?
UserDisplayName: Alexander Soare
ThreadText: # Are there any rules of thumb for having some idea of what capacity a neural network needs to have for a given problem?
  
  (Asked by: username_0 on 2020-02-24)
  
  To give an example. Let's just consider the MNIST dataset of handwritten digits. Here are some things which might have an impact on the optimum model capacity:
  
  * There are 10 output classes
  * The inputs are 28x28 grayscale pixels (I think this indirectly affects the model capacity. eg: if the inputs were 5x5 pixels, there wouldn't be much room for varying the way an 8 looks)
  
  **So, is there any way of knowing what the model capacity ought to be?** Even if it's not exact? Even if it's a qualitative understanding of the type "if X goes up, then Y goes down"?
  
  Just to accentuate what I mean when I say "not exact": I can already tell that a 100 variable model won't solve MNIST, so at least I have a lower bound. I'm also pretty sure that a 1,000,000,000 variable model is way more than needed. Of course, knowing a smaller range than that would be much more useful!
  
  **Tags:** neural-networks, computational-learning-theory, regularization, vc-dimension, capacity
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2020-03-05. Score: 4 | Accepted answer
  
  ## Theoretical results
  
  Rather than providing a rule of thumb (which can be misleading, so I am not a big fan of them), I will provide some theoretical results (the first one is also reported in paper [How many hidden layers and nodes?](http://dstath.users.uth.gr/papers/IJRS2009_Stathakis.pdf)), from which you may be able to derive your rules of thumb, depending on your problem, etc.
  
  ### Result 1
  
  The paper [Learning capability and storage capacity of two-hidden-layer feedforward networks](https://pdfs.semanticscholar.org/064f/1e85984b207c1eb3c53ac8b68037089b7a0b.pdf) proves that a 2-hidden layer feedforward
  network ($F$) with $$2 \sqrt{(m + 2)N} \ll N$$ hidden neurons can learn any $N$ distinct samples $D= \{ (x\_i, t\_i) \}\_{i=1}^N$ with an arbitrarily small error, where $m$ is the required number of output neurons. Conversely, a $F$ with $Q$ hidden neurons can store at least $\frac{Q^2}{4(m+2)}$ any distinct data $(x\_i, t\_i)$ with
  any desired precision.
  
  They suggest that a sufficient number of neurons in the first layer should be $\sqrt{(m + 2)N} + 2\sqrt{\frac{N}{m + 2}}$ and in the second layer should be $m\sqrt{\frac{N}{m + 2}}$. So, for example, if your dataset has size $N=10$ and you have $m=2$ output neurons, then you should have the first hidden layer with roughly 10 neurons and the second layer with roughly 4 neurons. (I haven't actually tried this!)
  
  However, these bounds are suited for fitting the training data (i.e. for overfitting), which isn't usually the goal, i.e. you want the network to generalize to unseen data.
  
  This result is strictly related to the universal approximation theorems, i.e. a network with a single hidden layer can, in theory, approximate any continuous function.
  
  ## Model selection, complexity control, and regularisation
  
  There are also the concepts of *model selection* and *complexity control*, and there are multiple related techniques that take into account the complexity of the model. The paper [Model complexity control and statistical learning theory](https://pdfs.semanticscholar.org/064f/1e85984b207c1eb3c53ac8b68037089b7a0b.pdf) (2002) may be useful. It is also important to note regularisation techniques can be thought of as controlling the complexity of the model [[1](https://www.nature.com/articles/s41467-020-14663-9)].
  
  ## Further reading
  
  You may also want to take a look at these related questions
  
  * [How to choose the number of hidden layers and nodes in a feedforward neural network?](https://stats.stackexchange.com/q/181/82135)
  * [How to estimate the capacity of a neural network?](https://ai.stackexchange.com/q/17870/2444)
  
  (I will be updating this answer, as I find more theoretical results or other useful info)
  
  
  ---
  
  ### Answer by username_1 on 2020-02-29. Score: 3
  
  This may sound counter intuitive but one of the biggest rules of thumb for model capacity in deep learning:
  
  **IT SHOULD OVERFIT**.
  
  Once you get a model to overfit, its easier to experiment with regularizations, module replacements, etc. But in general, it gives you a good starting ground.
  
  > **Comments:**
  > **username_2**: Although I understand your point, this sounds like a bad suggestion for several reasons. Ideally, you want to design the neural network before training it and not design by trial-and-error (which can be computationally prohibitive, if your task is difficult, you have a big dataset, big net, etc). What if the training data isn't really similar to the test data, even though this assumption is usually implicitly made? This rule of thumb could have been just a comment.  
  > **username_2**: Maybe you can rephrase this answer by saying that "In theory, if the model overfits the training data, it means that it has a sufficient capacity to model the training data, but this will require you to first train the model and try different configurations".  
  > **username_0**: Thanks for the tip. And @username_2 thanks for the clarity.  
  
  ---
  
  ### Answer by username_3 on 2020-02-28. Score: 0
  
  Personally, when I begin designing a machine learning model, I consider the following points:
  
  * My data: if I have simple images, like MNIST ones, or in general images with very low resolution, a very deep network is **not** required.
  * If my problem statement needs to learn a lot of features from each image, such as for the human face, I may need to learn eyes, nose, lips, expressions through their combinations, then I **need** a deep network with convolutional layers.
  * If I have time-series data, LSTM or GRU makes sense, but, I also consider recurrent setup when my data has high resolution, low count data points.
  
  The upper limit however may get decided by resources available on the computing device you are using for training.
  
  Hope this helps.
  
  
  ---
  
  