Answers.Body: <blockquote>
    <p>How can I decide the number of neurons in the FC layer before the softmax layer?</p>
  </blockquote>
  
  <p>Train different network architectures (with different numbers of neurons in the last FC layer). Use cross-validation - a set of data you have not trained on - to measure the performance of the network, using a metric that you have decided beforehand is a good proxy for your experiment goals. For instance, you might choose getting the highest classification accuracy as your goal - but might choose something different for an unbalanced dataset, because 90% accuracy is not meaningful.</p>
  
  <p>There is not usually a good reason to over-tune your network. Trying some variations with 1.5 x or 2 x geometric series (e.g. 5, 10, 20, 40 neurons in layer) is probably enough to find a good hidden layer size.</p>
  
  <blockquote>
    <p>Is it <span class="math-container">$N * 5$</span>, where <span class="math-container">$N$</span> is the number of classes?</p>
  </blockquote>
  
  <p>No, in general. That may work just fine for your problem though.</p>
  
  <blockquote>
    <p>Is there any documentation for deciding the number of neurons in the FC layer (before the softmax layer)</p>
  </blockquote>
  
  <p>Not really. The most direct thing to do is find solutions that work well for similar problems to yours, and adapt as necessary. So you can search for any similar problem domain and see what the researchers used there.</p>
  
  <p>There is not much theory to guide you here. However, if it helps with your intuition, more neurons can make a better fit to higher frequency variations in the target function you are learning, whilst more layers will mean better handling of complex function spaces (e.g. where rules about the mapping between input and output can be made according to combining simpler rules) - provided in both cases that you have enough training data that it is possible for the NN to learn the function approximately. These are not strictly defined traits of functions for machine learning, so many NN users will work with this intuitive view.</p>
  
  <p>If in doubt, assuming this is not an image, NLP or other well-studied problem, then I might just guess at e.g. 64 neurons per hidden layer, and try 1, 2, and 3 hidden layers as a starting point (all the same size). I cannot say if that will work for you and your problem, but it might help get past the "blank page effect" and start you training and testing some variations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-16T00:57:45.300Z
Answers.Text: Thanks. When I tried with CIFAR10, I got different accuracy for different number of neurons in the last hidden layer. I got the maximum accuracy when the number of neurons was 256. When I gave 100, it over fits early compared to 256. Also note that I am going to solve defect detection with images in a semi conductor Chip Manufacturing tool
Answers.UserDisplayName: Maanu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-16T08:01:53.293Z
Answers.Text: @Maanu: For defect detection I suspect you will not want a simple accuracy measure (I assume in production your defect rate is low, so the real world population is far from balanced). If you have trouble picking a metric, then perhaps ask here or on Data Science stack exchange. I think that Data Science is the better place to ask such practical questions about a specific problem domain.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-15T18:20:11.730Z
Answers.LastActivityDate: 2019-04-16T09:33:57.877Z
Answers.LastEditDate: 2019-04-16T09:33:57.877Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am solving a classification problem with CNN. The number of classes is 5. </p>
  
  <ol>
  <li>How can I decide the number of neurons in the FC layer before the softmax layer?</li>
  <li>Is it <span class="math-container">$N * 5$</span>, where <span class="math-container">$N$</span> is the number of classes?</li>
  <li>Is there any documentation for deciding the number of neurons in the FC layer (before SoftMax layer)</li>
  </ol>
  
Tags: <neural-networks><deep-learning><convolutional-neural-networks><ai-design>
Title: How do I choose the number of neurons in the fully-connected layer before the softmax layer?
UserDisplayName: Maanu
ThreadText: # How do I choose the number of neurons in the fully-connected layer before the softmax layer?
  
  (Asked by: username_0 on 2019-04-15)
  
  I am solving a classification problem with CNN. The number of classes is 5.
  
  1. How can I decide the number of neurons in the FC layer before the softmax layer?
  2. Is it $N \* 5$, where $N$ is the number of classes?
  3. Is there any documentation for deciding the number of neurons in the FC layer (before SoftMax layer)
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks, ai-design
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-04-15. Score: 1
  
  > How can I decide the number of neurons in the FC layer before the softmax layer?
  
  Train different network architectures (with different numbers of neurons in the last FC layer). Use cross-validation - a set of data you have not trained on - to measure the performance of the network, using a metric that you have decided beforehand is a good proxy for your experiment goals. For instance, you might choose getting the highest classification accuracy as your goal - but might choose something different for an unbalanced dataset, because 90% accuracy is not meaningful.
  
  There is not usually a good reason to over-tune your network. Trying some variations with 1.5 x or 2 x geometric series (e.g. 5, 10, 20, 40 neurons in layer) is probably enough to find a good hidden layer size.
  
  > Is it $N \* 5$, where $N$ is the number of classes?
  
  No, in general. That may work just fine for your problem though.
  
  > Is there any documentation for deciding the number of neurons in the FC layer (before the softmax layer)
  
  Not really. The most direct thing to do is find solutions that work well for similar problems to yours, and adapt as necessary. So you can search for any similar problem domain and see what the researchers used there.
  
  There is not much theory to guide you here. However, if it helps with your intuition, more neurons can make a better fit to higher frequency variations in the target function you are learning, whilst more layers will mean better handling of complex function spaces (e.g. where rules about the mapping between input and output can be made according to combining simpler rules) - provided in both cases that you have enough training data that it is possible for the NN to learn the function approximately. These are not strictly defined traits of functions for machine learning, so many NN users will work with this intuitive view.
  
  If in doubt, assuming this is not an image, NLP or other well-studied problem, then I might just guess at e.g. 64 neurons per hidden layer, and try 1, 2, and 3 hidden layers as a starting point (all the same size). I cannot say if that will work for you and your problem, but it might help get past the "blank page effect" and start you training and testing some variations.
  
  > **Comments:**
  > **username_0**: Thanks. When I tried with CIFAR10, I got different accuracy for different number of neurons in the last hidden layer. I got the maximum accuracy when the number of neurons was 256. When I gave 100, it over fits early compared to 256. Also note that I am going to solve defect detection with images in a semi conductor Chip Manufacturing tool  
  > **username_1**: @username_0: For defect detection I suspect you will not want a simple accuracy measure (I assume in production your defect rate is low, so the real world population is far from balanced). If you have trouble picking a metric, then perhaps ask here or on Data Science stack exchange. I think that Data Science is the better place to ask such practical questions about a specific problem domain.  
  
  ---
  
  