Answers.Body: <p>In your case the most probable explanation would be the case of overfitting. The model with too many hidden layers have lots of parameters. By means of all these parameters the model is remembering stuff from the training data itself instead of generalizing by learning the useful patterns.</p>
  <p>As a rule of thumb if you increase the number of hidden layers more and more at some point model would perform poorly. (I am assuming there is non-linearity in between. In case there is no non-linearity, it doesn't matter how much you stack, it would give the same result because it just boils down to one single layer).</p>
  <p>As an experiment, you can try to add regularization and you will see the model won't be performing that bad. Because now model is being punished for being too confident about the things it is remembering. As a result, it won't overfit to the training data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-11T07:51:58.187Z
Answers.LastActivityDate: 2021-04-11T07:51:58.187Z
Answers.UserDisplayName: user2736738
Answers.Body: <p>In general, yes.</p>
  <p>Stacking more layers and adding non-linearities will form a better function approximation (neural nets <strong>are</strong> basically function approximators), and when trained with the current regularization for each layer (such as L2 or L1) will cause your model to learn a better mapping, and hence generalize better.</p>
  <p>If you don't regularize, it will overfit.</p>
  <p>They are overparameterized, but why they don't overfit more (or in other words, why they generalize so well to unseen data) with increasing number of parameters is an effect that is even yet to be understood by the ML theory community [1]</p>
  <p>[1] - <a href="https://arxiv.org/abs/1806.11379" rel="nofollow noreferrer">https://arxiv.org/abs/1806.11379</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-11T18:31:23.867Z
Answers.LastActivityDate: 2021-04-11T18:31:23.867Z
Answers.UserDisplayName: Arjun Ashok
Body: <p>I trained different classification models using Keras with different numbers of hidden layers and the same number of neurons in each layer. What I found was the accuracy of the models decreased as the number of hidden layers increased However, the decrease was more significant in larger numbers of hidden layers. The accuracies refer to the test data and were obtained using k-fold=5. Also, no regularization was used. The following graph shows the accuracies of different models where the number of hidden layers changed while the rest of the parameters stayed the same (each model has 64 neurons in each hidden layer):</p>
  <p><a href="https://i.sstatic.net/JdZc9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JdZc9.png" alt="enter image description here" /></a></p>
  <p>My question is why is the drop in accuracy between 8 hidden layers and 16 hidden layers much greater than the drop between 1 hidden layer and 8 hidden layers, even though the difference in the number of hidden layers is the same (8).</p>
  
Tags: <deep-learning><hyperparameter-optimization><overfitting><hidden-layers><accuracy>
Title: Do larger numbers of hidden layers have a bigger effect on a classification model's accuracy?
UserDisplayName: Shonix3373
ThreadText: # Do larger numbers of hidden layers have a bigger effect on a classification model's accuracy?
  
  (Asked by: username_0 on 2021-04-11)
  
  I trained different classification models using Keras with different numbers of hidden layers and the same number of neurons in each layer. What I found was the accuracy of the models decreased as the number of hidden layers increased However, the decrease was more significant in larger numbers of hidden layers. The accuracies refer to the test data and were obtained using k-fold=5. Also, no regularization was used. The following graph shows the accuracies of different models where the number of hidden layers changed while the rest of the parameters stayed the same (each model has 64 neurons in each hidden layer):
  
  [![enter image description here](https://i.sstatic.net/JdZc9.png)](https://i.sstatic.net/JdZc9.png)
  
  My question is why is the drop in accuracy between 8 hidden layers and 16 hidden layers much greater than the drop between 1 hidden layer and 8 hidden layers, even though the difference in the number of hidden layers is the same (8).
  
  **Tags:** deep-learning, hyperparameter-optimization, overfitting, hidden-layers, accuracy
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-04-11. Score: 1
  
  In your case the most probable explanation would be the case of overfitting. The model with too many hidden layers have lots of parameters. By means of all these parameters the model is remembering stuff from the training data itself instead of generalizing by learning the useful patterns.
  
  As a rule of thumb if you increase the number of hidden layers more and more at some point model would perform poorly. (I am assuming there is non-linearity in between. In case there is no non-linearity, it doesn't matter how much you stack, it would give the same result because it just boils down to one single layer).
  
  As an experiment, you can try to add regularization and you will see the model won't be performing that bad. Because now model is being punished for being too confident about the things it is remembering. As a result, it won't overfit to the training data.
  
  
  ---
  
  ### Answer by username_2 on 2021-04-11. Score: 0
  
  In general, yes.
  
  Stacking more layers and adding non-linearities will form a better function approximation (neural nets **are** basically function approximators), and when trained with the current regularization for each layer (such as L2 or L1) will cause your model to learn a better mapping, and hence generalize better.
  
  If you don't regularize, it will overfit.
  
  They are overparameterized, but why they don't overfit more (or in other words, why they generalize so well to unseen data) with increasing number of parameters is an effect that is even yet to be understood by the ML theory community [1]
  
  [1] - <https://arxiv.org/abs/1806.11379>
  
  
  ---
  
  