Answers.Body: <p>In theory, deeper architectures can encode more information than shallower ones because they can perform more transformations of the input which results in better results at the output. The training is slower because back propagation is quite expensive, as you increase the depth, you increase the number of parameters and gradients that need to be computed.</p>
  
  <p>Another issue that you need to take into account is the effect of the activation function. Saturating functions like sigmoid and hyperbolic tangent result in very small gradients in their edges, other activation functions are just flat, eg. ReLU is flat on the negatives therefore, there is no error to propagate because the gradient is either very small (as in saturating functions) or zero. Batch Norm greatly assists in this operation because it collapses values in better ranges where the gradients aren't close to zero.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-29T11:19:39.790Z
Answers.LastActivityDate: 2019-10-29T20:52:53.777Z
Answers.LastEditDate: 2019-10-29T20:52:53.777Z
Answers.UserDisplayName: Dimitris Monroe
Answers.Body: <p>Those graphs do not disprove your 'vanishing gradient' theory. The deeper network may eventually do better than the shallower one, but it might take much longer to do it.</p>
  
  <p>Incidentally, the ReLU activation function was designed to mitigate the vanishing gradient problem. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-29T07:45:39.453Z
Answers.LastActivityDate: 2019-09-29T07:45:39.453Z
Answers.UserDisplayName: DrMcCleod
Body: <p>My understanding of the vanishing gradient problem in deep networks is that as backprop progresses through the layers the gradients become small, and thus training progresses slower. I'm having a hard time reconciling this understanding with images such as below where the losses for a deeper network are higher than for a shallower one. Should it not just take longer to complete each iteration, but still reach the same level if not higher of accuracy?</p>
  <p><a href="https://i.sstatic.net/y9HV7.png" rel="noreferrer"><img src="https://i.sstatic.net/y9HV7.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-30T03:36:43.820Z
Comments.Text: As a guess, it may be that when you have a large number of layers, the randomness of xavier initialisation may cause the numbers to explode/vanish. This is due to the fact that xavier init (and other inits) try to keep the variance at 1, but since they are random, the variance is more likely to be 1.03 or 0.97. With multiple layers stacked, this slight offset may begin to snowball into causing the numbers to explode/vanish.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-02T11:11:49.900Z
Comments.Text: Another guess: Consider that numbers in a computer are always just approximations. So computing the gradient layer by layer is a lossy process. This might make deeper layers not just slower to learn but actually introduce lower overall thresholds on possible accuracy.
Comments.UserDisplayName: BlindKungFuMaster
Tags: <deep-learning><training><backpropagation><architecture>
Title: Why do very deep non resnet architectures perform worse compared to shallower ones for the same iteration? Shouldn't they just train slower?
UserDisplayName: Intent Filters
ThreadText: # Why do very deep non resnet architectures perform worse compared to shallower ones for the same iteration? Shouldn't they just train slower?
  
  (Asked by: username_0 on 2019-09-28)
  
  My understanding of the vanishing gradient problem in deep networks is that as backprop progresses through the layers the gradients become small, and thus training progresses slower. I'm having a hard time reconciling this understanding with images such as below where the losses for a deeper network are higher than for a shallower one. Should it not just take longer to complete each iteration, but still reach the same level if not higher of accuracy?
  
  [![enter image description here](https://i.sstatic.net/y9HV7.png)](https://i.sstatic.net/y9HV7.png)
  
  **Tags:** deep-learning, training, backpropagation, architecture
  
  > **Comments:**
  > **username_1**: As a guess, it may be that when you have a large number of layers, the randomness of xavier initialisation may cause the numbers to explode/vanish. This is due to the fact that xavier init (and other inits) try to keep the variance at 1, but since they are random, the variance is more likely to be 1.03 or 0.97. With multiple layers stacked, this slight offset may begin to snowball into causing the numbers to explode/vanish.  
  > **username_2**: Another guess: Consider that numbers in a computer are always just approximations. So computing the gradient layer by layer is a lossy process. This might make deeper layers not just slower to learn but actually introduce lower overall thresholds on possible accuracy.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-10-29. Score: 1
  
  In theory, deeper architectures can encode more information than shallower ones because they can perform more transformations of the input which results in better results at the output. The training is slower because back propagation is quite expensive, as you increase the depth, you increase the number of parameters and gradients that need to be computed.
  
  Another issue that you need to take into account is the effect of the activation function. Saturating functions like sigmoid and hyperbolic tangent result in very small gradients in their edges, other activation functions are just flat, eg. ReLU is flat on the negatives therefore, there is no error to propagate because the gradient is either very small (as in saturating functions) or zero. Batch Norm greatly assists in this operation because it collapses values in better ranges where the gradients aren't close to zero.
  
  
  ---
  
  ### Answer by username_4 on 2019-09-29. Score: 0
  
  Those graphs do not disprove your 'vanishing gradient' theory. The deeper network may eventually do better than the shallower one, but it might take much longer to do it.
  
  Incidentally, the ReLU activation function was designed to mitigate the vanishing gradient problem.
  
  
  ---
  
  