Answers.Body: <h2>I/ Does it has any sense to apply backpropagation in these settings?</h2>
  <p>If I understand correctly, this question should be &quot;Can backpropagation return the gradient for every node of these networks?&quot;.</p>
  <p>It depends on your network <span class="math-container">$N$</span> if it is differentiable with all inputs <span class="math-container">$i_t$</span> so the backpropagation can be guaranteed that there are the gradient values at every node of the network.</p>
  <h2>II/ If yes, what happens to the gradients?</h2>
  <p>We deal with the vanishing or exploding gradient after we make sure that the gradient is <strong>exist</strong>. The reason for this phenomenon in a neural network can come various, some typical causes are:</p>
  <ol>
  <li><p>Very deep neural networks (a large number of layers):</p>
  <p>Assume a neural network with <span class="math-container">$n$</span>-th layers, the backpropagation is followed the Markov Chain's rule can be show as:</p>
  </li>
  </ol>
  <p><span class="math-container">$$\frac{\partial loss}{\partial w_t} = \frac{\partial loss}{\partial l_{n}} \times \frac{\partial l_{n}}{\partial l_{n-1}}\times...\times \frac{\partial l_{t+1}}{\partial l_{t}}\times \frac{\partial l_t}{\partial w_t}$$</span></p>
  <p>It's easy to see that the gradient is scaled exponentially for each layer, so if each gradient value is larger than 1, the gradient will become <span class="math-container">$\inf$</span> with <span class="math-container">$n\rightarrow \inf$</span> (exploding) and <span class="math-container">$0$</span> in vice versa.</p>
  <ol start="2">
  <li><p>Activation function:</p>
  <p>The image below shows the difference between different activation functions. It's easy to see that the logistic function such as Sigmoid or Tanh is limited in the range <span class="math-container">$[0,1]$</span> (in case sigmoid) or <span class="math-container">$[-1,1]$</span> in case Tanh. Therefore, if the output from a node of the neural network is larger than 2 or smaller than -2, the gradient will become nearly zero (vanishing) because the output is always the same.</p>
  </li>
  </ol>
  <p><a href="https://i.sstatic.net/Gsj3Y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Gsj3Y.png" alt="activation function" /></a></p>
  <p>Solution:</p>
  <ul>
  <li>Replace it by the simple function such as ReLU: <span class="math-container">$o_{t+1} = max(0, N(i_t))$</span>. However, the output from ReLU will be <span class="math-container">$0$</span> if it's lower than <span class="math-container">$0$</span>, so there are many variants of ReLU to solve this problem, you can find them easily by the keyword &quot;ReLU family&quot; on google (<a href="https://www.jefkine.com/deep/2016/08/08/initialization-of-deep-networks-case-of-rectifiers/" rel="nofollow noreferrer">example</a>).</li>
  </ul>
  <p>There are other methods or strategies to duel with vanishing/exploding gradient, it also depends on your model or your data. The answer can be more detailed if you give more information.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-11T02:12:30.777Z
Answers.LastActivityDate: 2021-09-10T22:34:15.657Z
Answers.LastEditDate: 2021-09-10T22:34:15.657Z
Answers.UserDisplayName: CuCaRot
Body: <p>I'm trying to train a Neural Network in a particular situation -- similar to a genetic algorithm domain as far as I know.
  I have to run a simulation with a length of <span class="math-container">$K$</span> steps.
  I have a neural network <span class="math-container">$N$</span> that at each time step is used to produce an output, so that:
  <span class="math-container">$$
  o_{t+1} = N(i_{t})
  $$</span>
  <span class="math-container">$i_t$</span> is a feature vector built upon <span class="math-container">$o_{t-1}$</span>, and <span class="math-container">$i_0$</span> is given.
  My ground-truth value is <span class="math-container">$o_k$</span>, namely the right value at the end of the simulation. So, I can evaluate the loss (e.g. MSE) only at the end of the simulation.
  Suppose to fix k to 3, the evaluation is: <span class="math-container">$N(N(N(i_0))$</span>
  because:
  <span class="math-container">$$
  o_1 = N(i_o) \\
  o_2 = N(i_1) \\
  o_3 = N(i_2)
  $$</span>
  So my questions are:</p>
  <ul>
  <li>does it have any sense to apply backpropagation in these settings?</li>
  <li>if yes, what happens to the gradients?</li>
  </ul>
  <p>Practically, in some simple situations, the backpropagation seems to work, but in others, the gradients explode or vanishes</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-30T15:14:33.410Z
Comments.Text: Is it possible that you could train it in steps, starting from short simulations and then working up to longer ones?
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-02T06:52:55.953Z
Comments.Text: In some case it is possible, but in the general case no..
Comments.UserDisplayName: gianluca aguzzi
Tags: <neural-networks><machine-learning>
Title: Backpropagation after N sequential input-output pass
UserDisplayName: gianluca aguzzi
ThreadText: # Backpropagation after N sequential input-output pass
  
  (Asked by: username_0 on 2021-07-30)
  
  I'm trying to train a Neural Network in a particular situation -- similar to a genetic algorithm domain as far as I know.
  I have to run a simulation with a length of $K$ steps.
  I have a neural network $N$ that at each time step is used to produce an output, so that:
  $$
  o\_{t+1} = N(i\_{t})
  $$
  $i\_t$ is a feature vector built upon $o\_{t-1}$, and $i\_0$ is given.
  My ground-truth value is $o\_k$, namely the right value at the end of the simulation. So, I can evaluate the loss (e.g. MSE) only at the end of the simulation.
  Suppose to fix k to 3, the evaluation is: $N(N(N(i\_0))$
  because:
  $$
  o\_1 = N(i\_o) \\
  o\_2 = N(i\_1) \\
  o\_3 = N(i\_2)
  $$
  So my questions are:
  
  * does it have any sense to apply backpropagation in these settings?
  * if yes, what happens to the gradients?
  
  Practically, in some simple situations, the backpropagation seems to work, but in others, the gradients explode or vanishes
  
  **Tags:** neural-networks, machine-learning
  
  > **Comments:**
  > **username_1**: Is it possible that you could train it in steps, starting from short simulations and then working up to longer ones?  
  > **username_0**: In some case it is possible, but in the general case no..  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-08-11. Score: 1
  
  ## I/ Does it has any sense to apply backpropagation in these settings?
  
  If I understand correctly, this question should be "Can backpropagation return the gradient for every node of these networks?".
  
  It depends on your network $N$ if it is differentiable with all inputs $i\_t$ so the backpropagation can be guaranteed that there are the gradient values at every node of the network.
  
  ## II/ If yes, what happens to the gradients?
  
  We deal with the vanishing or exploding gradient after we make sure that the gradient is **exist**. The reason for this phenomenon in a neural network can come various, some typical causes are:
  
  1. Very deep neural networks (a large number of layers):
  
     Assume a neural network with $n$-th layers, the backpropagation is followed the Markov Chain's rule can be show as:
  
  $$\frac{\partial loss}{\partial w\_t} = \frac{\partial loss}{\partial l\_{n}} \times \frac{\partial l\_{n}}{\partial l\_{n-1}}\times...\times \frac{\partial l\_{t+1}}{\partial l\_{t}}\times \frac{\partial l\_t}{\partial w\_t}$$
  
  It's easy to see that the gradient is scaled exponentially for each layer, so if each gradient value is larger than 1, the gradient will become $\inf$ with $n\rightarrow \inf$ (exploding) and $0$ in vice versa.
  
  2. Activation function:
  
     The image below shows the difference between different activation functions. It's easy to see that the logistic function such as Sigmoid or Tanh is limited in the range $[0,1]$ (in case sigmoid) or $[-1,1]$ in case Tanh. Therefore, if the output from a node of the neural network is larger than 2 or smaller than -2, the gradient will become nearly zero (vanishing) because the output is always the same.
  
  [![activation function](https://i.sstatic.net/Gsj3Y.png)](https://i.sstatic.net/Gsj3Y.png)
  
  Solution:
  
  * Replace it by the simple function such as ReLU: $o\_{t+1} = max(0, N(i\_t))$. However, the output from ReLU will be $0$ if it's lower than $0$, so there are many variants of ReLU to solve this problem, you can find them easily by the keyword "ReLU family" on google ([example](https://www.jefkine.com/deep/2016/08/08/initialization-of-deep-networks-case-of-rectifiers/)).
  
  There are other methods or strategies to duel with vanishing/exploding gradient, it also depends on your model or your data. The answer can be more detailed if you give more information.
  
  
  ---
  
  