<h2>I/ Does it has any sense to apply backpropagation in these settings?</h2>
<p>If I understand correctly, this question should be &quot;Can backpropagation return the gradient for every node of these networks?&quot;.</p>
<p>It depends on your network <span class="math-container">$N$</span> if it is differentiable with all inputs <span class="math-container">$i_t$</span> so the backpropagation can be guaranteed that there are the gradient values at every node of the network.</p>
<h2>II/ If yes, what happens to the gradients?</h2>
<p>We deal with the vanishing or exploding gradient after we make sure that the gradient is <strong>exist</strong>. The reason for this phenomenon in a neural network can come various, some typical causes are:</p>
<ol>
<li><p>Very deep neural networks (a large number of layers):</p>
<p>Assume a neural network with <span class="math-container">$n$</span>-th layers, the backpropagation is followed the Markov Chain's rule can be show as:</p>
</li>
</ol>
<p><span class="math-container">$$\frac{\partial loss}{\partial w_t} = \frac{\partial loss}{\partial l_{n}} \times \frac{\partial l_{n}}{\partial l_{n-1}}\times...\times \frac{\partial l_{t+1}}{\partial l_{t}}\times \frac{\partial l_t}{\partial w_t}$$</span></p>
<p>It's easy to see that the gradient is scaled exponentially for each layer, so if each gradient value is larger than 1, the gradient will become <span class="math-container">$\inf$</span> with <span class="math-container">$n\rightarrow \inf$</span> (exploding) and <span class="math-container">$0$</span> in vice versa.</p>
<ol start="2">
<li><p>Activation function:</p>
<p>The image below shows the difference between different activation functions. It's easy to see that the logistic function such as Sigmoid or Tanh is limited in the range <span class="math-container">$[0,1]$</span> (in case sigmoid) or <span class="math-container">$[-1,1]$</span> in case Tanh. Therefore, if the output from a node of the neural network is larger than 2 or smaller than -2, the gradient will become nearly zero (vanishing) because the output is always the same.</p>
</li>
</ol>
<p><a href="https://i.sstatic.net/Gsj3Y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Gsj3Y.png" alt="activation function" /></a></p>
<p>Solution:</p>
<ul>
<li>Replace it by the simple function such as ReLU: <span class="math-container">$o_{t+1} = max(0, N(i_t))$</span>. However, the output from ReLU will be <span class="math-container">$0$</span> if it's lower than <span class="math-container">$0$</span>, so there are many variants of ReLU to solve this problem, you can find them easily by the keyword &quot;ReLU family&quot; on google (<a href="https://www.jefkine.com/deep/2016/08/08/initialization-of-deep-networks-case-of-rectifiers/" rel="nofollow noreferrer">example</a>).</li>
</ul>
<p>There are other methods or strategies to duel with vanishing/exploding gradient, it also depends on your model or your data. The answer can be more detailed if you give more information.</p>

CC BY-SA 4.0
2021-08-11T02:12:30.777Z
2021-09-10T22:34:15.657Z
2021-09-10T22:34:15.657Z
CuCaRot
<p>I'm trying to train a Neural Network in a particular situation -- similar to a genetic algorithm domain as far as I know.
I have to run a simulation with a length of <span class="math-container">$K$</span> steps.
I have a neural network <span class="math-container">$N$</span> that at each time step is used to produce an output, so that:
<span class="math-container">$$
o_{t+1} = N(i_{t})
$$</span>
<span class="math-container">$i_t$</span> is a feature vector built upon <span class="math-container">$o_{t-1}$</span>, and <span class="math-container">$i_0$</span> is given.
My ground-truth value is <span class="math-container">$o_k$</span>, namely the right value at the end of the simulation. So, I can evaluate the loss (e.g. MSE) only at the end of the simulation.
Suppose to fix k to 3, the evaluation is: <span class="math-container">$N(N(N(i_0))$</span>
because:
<span class="math-container">$$
o_1 = N(i_o) \\
o_2 = N(i_1) \\
o_3 = N(i_2)
$$</span>
So my questions are:</p>
<ul>
<li>does it have any sense to apply backpropagation in these settings?</li>
<li>if yes, what happens to the gradients?</li>
</ul>
<p>Practically, in some simple situations, the backpropagation seems to work, but in others, the gradients explode or vanishes</p>

CC BY-SA 4.0
2021-07-30T15:14:33.410Z
Is it possible that you could train it in steps, starting from short simulations and then working up to longer ones?
Criticize SE actions means ban
CC BY-SA 4.0
2021-08-02T06:52:55.953Z
In some case it is possible, but in the general case no..
gianluca aguzzi
<neural-networks><machine-learning>
Backpropagation after N sequential input-output pass
gianluca aguzzi
# Backpropagation after N sequential input-output pass

(Asked by: username_0 on 2021-07-30)

I'm trying to train a Neural Network in a particular situation -- similar to a genetic algorithm domain as far as I know.
I have to run a simulation with a length of $K$ steps.
I have a neural network $N$ that at each time step is used to produce an output, so that:
$$
o\_{t+1} = N(i\_{t})
$$
$i\_t$ is a feature vector built upon $o\_{t-1}$, and $i\_0$ is given.
My ground-truth value is $o\_k$, namely the right value at the end of the simulation. So, I can evaluate the loss (e.g. MSE) only at the end of the simulation.
Suppose to fix k to 3, the evaluation is: $N(N(N(i\_0))$
because:
$$
o\_1 = N(i\_o) \\
o\_2 = N(i\_1) \\
o\_3 = N(i\_2)
$$
So my questions are:

* does it have any sense to apply backpropagation in these settings?
* if yes, what happens to the gradients?

Practically, in some simple situations, the backpropagation seems to work, but in others, the gradients explode or vanishes

**Tags:** neural-networks, machine-learning

> **Comments:**
> **username_1**: Is it possible that you could train it in steps, starting from short simulations and then working up to longer ones?  
> **username_0**: In some case it is possible, but in the general case no..  

---

## 1 answers

### Answer by username_2 on 2021-08-11. Score: 1

## I/ Does it has any sense to apply backpropagation in these settings?

If I understand correctly, this question should be "Can backpropagation return the gradient for every node of these networks?".

It depends on your network $N$ if it is differentiable with all inputs $i\_t$ so the backpropagation can be guaranteed that there are the gradient values at every node of the network.

## II/ If yes, what happens to the gradients?

We deal with the vanishing or exploding gradient after we make sure that the gradient is **exist**. The reason for this phenomenon in a neural network can come various, some typical causes are:

1. Very deep neural networks (a large number of layers):

   Assume a neural network with $n$-th layers, the backpropagation is followed the Markov Chain's rule can be show as:

$$\frac{\partial loss}{\partial w\_t} = \frac{\partial loss}{\partial l\_{n}} \times \frac{\partial l\_{n}}{\partial l\_{n-1}}\times...\times \frac{\partial l\_{t+1}}{\partial l\_{t}}\times \frac{\partial l\_t}{\partial w\_t}$$

It's easy to see that the gradient is scaled exponentially for each layer, so if each gradient value is larger than 1, the gradient will become $\inf$ with $n\rightarrow \inf$ (exploding) and $0$ in vice versa.

2. Activation function:

   The image below shows the difference between different activation functions. It's easy to see that the logistic function such as Sigmoid or Tanh is limited in the range $[0,1]$ (in case sigmoid) or $[-1,1]$ in case Tanh. Therefore, if the output from a node of the neural network is larger than 2 or smaller than -2, the gradient will become nearly zero (vanishing) because the output is always the same.

[![activation function](https://i.sstatic.net/Gsj3Y.png)](https://i.sstatic.net/Gsj3Y.png)

Solution:

* Replace it by the simple function such as ReLU: $o\_{t+1} = max(0, N(i\_t))$. However, the output from ReLU will be $0$ if it's lower than $0$, so there are many variants of ReLU to solve this problem, you can find them easily by the keyword "ReLU family" on google ([example](https://www.jefkine.com/deep/2016/08/08/initialization-of-deep-networks-case-of-rectifiers/)).

There are other methods or strategies to duel with vanishing/exploding gradient, it also depends on your model or your data. The answer can be more detailed if you give more information.


---

