Answers.Body: <p>Your understanding is totally correct. The <em>chain rule</em> is defined as the product of derivatives, and as you well mention, from the mathematical point of view four scenarios can happen (you can visualize them <a href="https://www.desmos.com/calculator/civpvqvkww" rel="nofollow noreferrer">here</a>):</p>
  <ol>
  <li>If the terms are in <span class="math-container">$(-1,1)$</span>, in their limit they tend to 0.</li>
  <li>If they are all 1, they stay at 1.</li>
  <li>If they are all -1, they alternate between -1 and 1.</li>
  <li>If they are greater than 1, they tend to <span class="math-container">$\infty$</span>. If they are smaller than -1 they tend to <span class="math-container">$-\infty$</span>.</li>
  </ol>
  <p>In practice, however, cases 1. and 4. are the most common, and most of the strategies (e.g. resnet, lstm) are designed to tackle this problem as you probably well know already.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-18T13:02:23.493Z
Answers.LastActivityDate: 2021-11-18T13:02:23.493Z
Answers.UserDisplayName: Jzbach
Body: <p>I am asking this question from the mathematical perspective of the vanishing and exploding gradient problems that we face generally during training deep neural networks.</p>
  <p>The chain rule of differentiation for a composite function can be expressed roughly as follows:</p>
  <p><span class="math-container">$$\dfrac{d}{dx} (f_1(f_2(f_3(\cdots f_n(x))))) = \dfrac{df_n}{dx} \dfrac{df_{n-1}}{dy_1} \dfrac{df_{n-2}}{dy_2} \cdots \dfrac{df_1}{dy_{n-1}}$$</span></p>
  <p>We know that multilayer perceptrons are composite functions of layer functions. So, if layers are increasing, then the gradient terms to multiply will increase on the right-hand side.</p>
  <p>If all the gradient terms on the right-hand side are between 0 and 1 then the gradient will become less and less if layers keep on increasing the layers. This phenomenon is called the vanishing gradient problem. Similarly, if all the gradient terms on the right-hand side are greater than 1. Then the product will become more and more. This phenomenon is called exploding gradient problem.</p>
  <p>Since it is customary to use the same activation function across all the layers in deep neural networks, all the gradients on the right hand behave in a similar manner, i.e. either most of the gradient terms on the right-hand side fall between 0 and 1 or greater than one, which causes either vanishing gradient or exploding gradient problem.</p>
  <p>Do my mathematical interpretation of the vanishing and exploding gradient problem true?  Am I missing anything?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-24T00:57:07.773Z
Comments.Text: It looks correct to me. At least, that's my understanding of vanishing/exploding gradients too.
Comments.UserDisplayName: SpiderRico
Tags: <math><gradient><vanishing-gradient-problem><exploding-gradient-problem>
Title: Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?
UserDisplayName: hanugm
ThreadText: # Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?
  
  (Asked by: username_0 on 2021-09-24)
  
  I am asking this question from the mathematical perspective of the vanishing and exploding gradient problems that we face generally during training deep neural networks.
  
  The chain rule of differentiation for a composite function can be expressed roughly as follows:
  
  $$\dfrac{d}{dx} (f\_1(f\_2(f\_3(\cdots f\_n(x))))) = \dfrac{df\_n}{dx} \dfrac{df\_{n-1}}{dy\_1} \dfrac{df\_{n-2}}{dy\_2} \cdots \dfrac{df\_1}{dy\_{n-1}}$$
  
  We know that multilayer perceptrons are composite functions of layer functions. So, if layers are increasing, then the gradient terms to multiply will increase on the right-hand side.
  
  If all the gradient terms on the right-hand side are between 0 and 1 then the gradient will become less and less if layers keep on increasing the layers. This phenomenon is called the vanishing gradient problem. Similarly, if all the gradient terms on the right-hand side are greater than 1. Then the product will become more and more. This phenomenon is called exploding gradient problem.
  
  Since it is customary to use the same activation function across all the layers in deep neural networks, all the gradients on the right hand behave in a similar manner, i.e. either most of the gradient terms on the right-hand side fall between 0 and 1 or greater than one, which causes either vanishing gradient or exploding gradient problem.
  
  Do my mathematical interpretation of the vanishing and exploding gradient problem true? Am I missing anything?
  
  **Tags:** math, gradient, vanishing-gradient-problem, exploding-gradient-problem
  
  > **Comments:**
  > **username_1**: It looks correct to me. At least, that's my understanding of vanishing/exploding gradients too.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-11-18. Score: 2 | Accepted answer
  
  Your understanding is totally correct. The *chain rule* is defined as the product of derivatives, and as you well mention, from the mathematical point of view four scenarios can happen (you can visualize them [here](https://www.desmos.com/calculator/civpvqvkww)):
  
  1. If the terms are in $(-1,1)$, in their limit they tend to 0.
  2. If they are all 1, they stay at 1.
  3. If they are all -1, they alternate between -1 and 1.
  4. If they are greater than 1, they tend to $\infty$. If they are smaller than -1 they tend to $-\infty$.
  
  In practice, however, cases 1. and 4. are the most common, and most of the strategies (e.g. resnet, lstm) are designed to tackle this problem as you probably well know already.
  
  
  ---
  
  