Answers.Body: <p>You probably got the back propagation wrong. I have done a test on the accuracy on adding an extra layer and the accuracy went up from 94% to 96% for me. See this for details:</p>
  
  <p><a href="https://colab.research.google.com/drive/17kAJ2KJ36grG9sz-KW10fZCQW9i2Tf2c" rel="nofollow noreferrer">https://colab.research.google.com/drive/17kAJ2KJ36grG9sz-KW10fZCQW9i2Tf2c</a></p>
  
  <p>To run the notebook click <code>Open in playground</code> and run the code. There is a commented line which add 1 extra layer. The syntax should be easy to understand even though it is in python. </p>
  
  <p>For back propagation, you can try to see this python implementation of multi layer perceptron backpropagation.</p>
  
  <p><a href="https://github.com/enggen/Deep-Learning-Coursera/blob/master/Neural%20Networks%20and%20Deep%20Learning/Building%20your%20Deep%20Neural%20Network%20-%20Step%20by%20Step.ipynb" rel="nofollow noreferrer">https://github.com/enggen/Deep-Learning-Coursera/blob/master/Neural%20Networks%20and%20Deep%20Learning/Building%20your%20Deep%20Neural%20Network%20-%20Step%20by%20Step.ipynb</a></p>
  
  <p>A network will not usually decrease it's accuracy by almost a half in normal scenario when you add an extra layer, though it is possible to have the network decrease accuracy when you add an extra layer due to overfitting. Though if this happen the performance drop won't be that dramatic.</p>
  
  <p>Hope I can help you.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-04T23:51:03.243Z
Answers.Text: Thanks Clement! I’ve accepted this as you are the first to write an answer, and very detailed as well :) I’m looking into my backpropagation code now and will have a look at your code as well… Thanks
Answers.UserDisplayName: Chris
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-04T23:53:28.300Z
Answers.Text: Also, I implemented Xavier’s weight initialization  and Kaiming’s weight initialization… And unfortunately neither fix the issue, so it really does seem to be the backpropagation causing trouble
Answers.UserDisplayName: Chris
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-05T04:24:13.033Z
Answers.Text: Weight Initialization shouldn't be the cause of the problem, as normal random initialization will already be good, the Xavier initialization should only increase training speed.
Answers.UserDisplayName: Clement
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-04T11:48:53.920Z
Answers.LastActivityDate: 2020-01-04T11:48:53.920Z
Answers.UserDisplayName: Clement
Body: <p>I've been using several resources to implement my own artificial neural network package in C++.</p>
  
  <p>Among some of the resources I've been using are </p>
  
  <p><a href="https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/" rel="nofollow noreferrer">https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/</a></p>
  
  <p><a href="https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/" rel="nofollow noreferrer">https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/</a></p>
  
  <p><a href="https://cs.stanford.edu/people/karpathy/convnetjs/intro.html" rel="nofollow noreferrer">https://cs.stanford.edu/people/karpathy/convnetjs/intro.html</a>, </p>
  
  <p>as well as several others.</p>
  
  <p>My code manages to replicate the results in the first two resources exactly. However, these are fairly simple networks in terms of depth. Hence the following (detailed) question:</p>
  
  <p>For my implementation, I've been working with the MNIST Database of handwritten digits (<a href="http://yann.lecun.com/exdb/mnist/" rel="nofollow noreferrer">http://yann.lecun.com/exdb/mnist/</a>).</p>
  
  <p>Using the ANN package I wrote, I have created a simple ANN with 784 input neurons, one hidden layer with 16 neurons, as well as an output layer with ten neurons. I have implemented ReLU on the hidden layer and the ouput layer, as well as a softmax on the output layer to get probabilities.The weights and biases are each individiually initialized to random values in the range [-1,1]</p>
  
  <p>So the network is 784x16x10.</p>
  
  <p>My backpropagation incorporates weight gradient and bias gradient logic.</p>
  
  <p>With this configuration, I repeatedly get about a 90% hit rate with a total average cost of ~0.07 on the MNIST training set comprising 60,000 digits, and a slightly higher hit rate of ~92.5% on the test set comprising 10,000 digits.</p>
  
  <p>For my first implementation of an ANN, I am pretty happy with that. However, my next thought was: </p>
  
  <p><strong>"If I add another hidden layer, I should get even better results...?".</strong></p>
  
  <p>So I created another artificial network with the same configuration, except for the addition of another hidden layer of 16 neurons, which I also run through a reLU. So this network is 784x16x16x10.</p>
  
  <p>On this ANN, I get significantly worse results. The hit rate on the training set repeatedly comes out at ~45% with a total average error of ~0.35, and on the test set I also only get about 45%.</p>
  
  <p>This leads me to either one or both of the following conclusions:</p>
  
  <p><strong>A) My implementation of the ANN in C++ is somehow faulty. If so, my bet would be it is somewhere in the backpropagation, as I am not 100% certain my weight gradient and bias gradient calculation is correct for any layers before the last hidden layer.</strong></p>
  
  <p><strong>B) This is an expected effect. Something about adding another layer makes the ANN not suitable for this (digit classification) kind of problem.</strong></p>
  
  <p>Of course, A, B, or A and B could be true.</p>
  
  <p><strong>Could someone with more experience than me give me some input, especially on whether B) is true or not?</strong></p>
  
  <p>If B) is not true, then I know I have to look at my code again.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-03T10:17:49.553Z
Comments.Text: Network should learn fine with additional layer, but the problem might not be in the backpropagation code either. There are other details to consider. Are you learning for enough epochs? Deeper models generally need more time to train. Your initial weight range $[-1, 1]$ might be too large, you can try researching about initialization strategies (e.g. Xavier initialization). Are you shuffling your data while learning? There are probably some other things to consider that I didn't mention. You can try using available deep learning software (e.g. Tensorflow) to compare results.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-04T03:21:12.990Z
Comments.Text: Thanks Brale! I will have a look at those as well. I’m currently not shuffling my data, and I was also wondering about Xavier’s Weight Initialization. I’ll try and see if that improves results.
Comments.UserDisplayName: Chris
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-04T03:24:02.023Z
Comments.Text: By the way,@Brale_, if you say B) is not true you could make your comment into an answer and I’ll accept it.
Comments.UserDisplayName: Chris
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-04T11:50:21.263Z
Comments.Text: I tried using keras to compare the results and it's in a colab notebook @Chris hope I can help you. https://colab.research.google.com/drive/17kAJ2KJ36grG9sz-KW10fZCQW9i2Tf2c
Comments.UserDisplayName: Clement
Tags: <image-recognition><hidden-layers><multilayer-perceptrons><c++>
Title: Is it expected that adding an additional hidden layer to my 3-layer ANN reduces accuracy significantly?
UserDisplayName: Chris
ThreadText: # Is it expected that adding an additional hidden layer to my 3-layer ANN reduces accuracy significantly?
  
  (Asked by: username_0 on 2020-01-03)
  
  I've been using several resources to implement my own artificial neural network package in C++.
  
  Among some of the resources I've been using are
  
  <https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/>
  
  <https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/>
  
  <https://cs.stanford.edu/people/karpathy/convnetjs/intro.html>,
  
  as well as several others.
  
  My code manages to replicate the results in the first two resources exactly. However, these are fairly simple networks in terms of depth. Hence the following (detailed) question:
  
  For my implementation, I've been working with the MNIST Database of handwritten digits (<http://yann.lecun.com/exdb/mnist/>).
  
  Using the ANN package I wrote, I have created a simple ANN with 784 input neurons, one hidden layer with 16 neurons, as well as an output layer with ten neurons. I have implemented ReLU on the hidden layer and the ouput layer, as well as a softmax on the output layer to get probabilities.The weights and biases are each individiually initialized to random values in the range [-1,1]
  
  So the network is 784x16x10.
  
  My backpropagation incorporates weight gradient and bias gradient logic.
  
  With this configuration, I repeatedly get about a 90% hit rate with a total average cost of ~0.07 on the MNIST training set comprising 60,000 digits, and a slightly higher hit rate of ~92.5% on the test set comprising 10,000 digits.
  
  For my first implementation of an ANN, I am pretty happy with that. However, my next thought was:
  
  **"If I add another hidden layer, I should get even better results...?".**
  
  So I created another artificial network with the same configuration, except for the addition of another hidden layer of 16 neurons, which I also run through a reLU. So this network is 784x16x16x10.
  
  On this ANN, I get significantly worse results. The hit rate on the training set repeatedly comes out at ~45% with a total average error of ~0.35, and on the test set I also only get about 45%.
  
  This leads me to either one or both of the following conclusions:
  
  **A) My implementation of the ANN in C++ is somehow faulty. If so, my bet would be it is somewhere in the backpropagation, as I am not 100% certain my weight gradient and bias gradient calculation is correct for any layers before the last hidden layer.**
  
  **B) This is an expected effect. Something about adding another layer makes the ANN not suitable for this (digit classification) kind of problem.**
  
  Of course, A, B, or A and B could be true.
  
  **Could someone with more experience than me give me some input, especially on whether B) is true or not?**
  
  If B) is not true, then I know I have to look at my code again.
  
  **Tags:** image-recognition, hidden-layers, multilayer-perceptrons, c++
  
  > **Comments:**
  > **username_1**: Network should learn fine with additional layer, but the problem might not be in the backpropagation code either. There are other details to consider. Are you learning for enough epochs? Deeper models generally need more time to train. Your initial weight range $[-1, 1]$ might be too large, you can try researching about initialization strategies (e.g. Xavier initialization). Are you shuffling your data while learning? There are probably some other things to consider that I didn't mention. You can try using available deep learning software (e.g. Tensorflow) to compare results.  
  > **username_0**: Thanks username_1! I will have a look at those as well. I’m currently not shuffling my data, and I was also wondering about Xavier’s Weight Initialization. I’ll try and see if that improves results.  
  > **username_0**: By the way,@Brale_, if you say B) is not true you could make your comment into an answer and I’ll accept it.  
  > **username_2**: I tried using keras to compare the results and it's in a colab notebook @username_0 hope I can help you. https://colab.research.google.com/drive/17kAJ2KJ36grG9sz-KW10fZCQW9i2Tf2c  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-01-04. Score: 2 | Accepted answer
  
  You probably got the back propagation wrong. I have done a test on the accuracy on adding an extra layer and the accuracy went up from 94% to 96% for me. See this for details:
  
  <https://colab.research.google.com/drive/17kAJ2KJ36grG9sz-KW10fZCQW9i2Tf2c>
  
  To run the notebook click `Open in playground` and run the code. There is a commented line which add 1 extra layer. The syntax should be easy to understand even though it is in python.
  
  For back propagation, you can try to see this python implementation of multi layer perceptron backpropagation.
  
  <https://github.com/enggen/Deep-Learning-Coursera/blob/master/Neural%20Networks%20and%20Deep%20Learning/Building%20your%20Deep%20Neural%20Network%20-%20Step%20by%20Step.ipynb>
  
  A network will not usually decrease it's accuracy by almost a half in normal scenario when you add an extra layer, though it is possible to have the network decrease accuracy when you add an extra layer due to overfitting. Though if this happen the performance drop won't be that dramatic.
  
  Hope I can help you.
  
  > **Comments:**
  > **username_0**: Thanks username_2! I’ve accepted this as you are the first to write an answer, and very detailed as well :) I’m looking into my backpropagation code now and will have a look at your code as well… Thanks  
  > **username_0**: Also, I implemented Xavier’s weight initialization  and Kaiming’s weight initialization… And unfortunately neither fix the issue, so it really does seem to be the backpropagation causing trouble  
  > **username_2**: Weight Initialization shouldn't be the cause of the problem, as normal random initialization will already be good, the Xavier initialization should only increase training speed.  
  
  ---
  
  