Answers.Body: <p>The echo state condition states that differences in the input sequence results in separate trajectories of reservoir states.</p>
  <p>Which means that when you make the reservoir large enough any difference in the input sequence result in a linearly separable difference in the state space.</p>
  <p>This is quite similar to the Kerneltrick in e.g. Support Vector machines where the data becomes linearly separable in the feature space. In reservoir computing the reservoir is in a way a random feature space.</p>
  <p>To sum up, the echo state condition ensures that the signal becomes linearly separable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-01T16:51:23.227Z
Answers.Text: Can you state this intuition in the form of a theorem? It is not clear to me how you get linear separability just from echo state (or even what "linearly separable" really means in the context of sequence-to-sequence transformations).
Answers.UserDisplayName: Jack M
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-04T17:36:24.647Z
Answers.Text: In this articel (http://www.scholarpedia.org/article/Echo_state_network) Jaeger writes: "ESP states that the reservoir will asymptotically wash out any information from initial conditions." This "fading memory property" was formalized by Boyd et al (Fading Memory and the Problem of Approximating Nonlinear Operators with Volterra Series, 1985): for all epsilon there exists delta>0 and T >0 such that |F(v)(t=0)-F(u)(t=0)| < epsilon with |u(t) - v(t)| < delta for all t \in [-T,0].
Answers.UserDisplayName: claire_fontaine
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-04T17:46:31.410Z
Answers.Text: where F is some time-invariant, causal Operator (the reservoir in our case).
Answers.UserDisplayName: claire_fontaine
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-01T16:50:06.833Z
Answers.LastActivityDate: 2020-12-01T16:50:06.833Z
Answers.UserDisplayName: claire_fontaine
Body: <p>Slightly generalizing the definition in <a href="https://www.researchgate.net/profile/Herbert_Jaeger3/publication/215385037_The_echo_state_approach_to_analysing_and_training_recurrent_neural_networks-with_an_erratum_note%27/links/566a003508ae62b05f027be3/The-echo-state-approach-to-analysing-and-training-recurrent-neural-networks-with-an-erratum-note.pdf?origin=publication_detail" rel="nofollow noreferrer">Jaeger 2001</a>, let's define a <strong>reservoir system</strong> to be any system of the form</p>
  <p><span class="math-container">$$h_{t}=f(h_{t-1}, x_t)$$</span>
  <span class="math-container">$$y_t=g(Wh_t)$$</span></p>
  <p>where <span class="math-container">$f$</span> and <span class="math-container">$g$</span> are fixed and <span class="math-container">$W$</span> is a learnable weight matrix. The idea is we feed a sequence of inputs <span class="math-container">$x_t$</span> into the system, which has some fixed initial state <span class="math-container">$h_0$</span>, and thereby generate the sequence of outputs <span class="math-container">$y_t$</span>. <span class="math-container">$f$</span> is fixed (for example, a randomly generated RNN) we can then attempt to learn <span class="math-container">$W$</span> in some way in order to get the system to have the behavior that we want.</p>
  <p>Now we add the <strong>echo state condition</strong>: the system has the echo state condition iff for any left-infinite sequence <span class="math-container">$...x_{-3}, x_{-2}, x_{-1}, x_0$</span>, there is only one sequence of states <span class="math-container">$h_t$</span> consistent with this input sequence.</p>
  <p>Seen from this perspective, any training procedure that could be applied to an echo state system could be applied to a generic reservoir system. So <strong>what do we get out of the echo state condition</strong>? Is there some reason to think echo state systems will generalize better, or be more quickly trainable? Jaeger does not seem to attempt to argue in this direction, he just describes how to train an ESN, but as I've said, nothing about these training methods seems to require the echo state property.</p>
  
Tags: <neural-networks><echo-state-network><reservoir-computing>
Title: What do echo state networks give us over a generic RNN resevoir?
UserDisplayName: Jack M
ThreadText: # What do echo state networks give us over a generic RNN resevoir?
  
  (Asked by: username_0 on 2020-11-28)
  
  Slightly generalizing the definition in [Jaeger 2001](https://www.researchgate.net/profile/Herbert_Jaeger3/publication/215385037_The_echo_state_approach_to_analysing_and_training_recurrent_neural_networks-with_an_erratum_note%27/links/566a003508ae62b05f027be3/The-echo-state-approach-to-analysing-and-training-recurrent-neural-networks-with-an-erratum-note.pdf?origin=publication_detail), let's define a **reservoir system** to be any system of the form
  
  $$h\_{t}=f(h\_{t-1}, x\_t)$$
  $$y\_t=g(Wh\_t)$$
  
  where $f$ and $g$ are fixed and $W$ is a learnable weight matrix. The idea is we feed a sequence of inputs $x\_t$ into the system, which has some fixed initial state $h\_0$, and thereby generate the sequence of outputs $y\_t$. $f$ is fixed (for example, a randomly generated RNN) we can then attempt to learn $W$ in some way in order to get the system to have the behavior that we want.
  
  Now we add the **echo state condition**: the system has the echo state condition iff for any left-infinite sequence $...x\_{-3}, x\_{-2}, x\_{-1}, x\_0$, there is only one sequence of states $h\_t$ consistent with this input sequence.
  
  Seen from this perspective, any training procedure that could be applied to an echo state system could be applied to a generic reservoir system. So **what do we get out of the echo state condition**? Is there some reason to think echo state systems will generalize better, or be more quickly trainable? Jaeger does not seem to attempt to argue in this direction, he just describes how to train an ESN, but as I've said, nothing about these training methods seems to require the echo state property.
  
  **Tags:** neural-networks, echo-state-network, reservoir-computing
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-01. Score: 1
  
  The echo state condition states that differences in the input sequence results in separate trajectories of reservoir states.
  
  Which means that when you make the reservoir large enough any difference in the input sequence result in a linearly separable difference in the state space.
  
  This is quite similar to the Kerneltrick in e.g. Support Vector machines where the data becomes linearly separable in the feature space. In reservoir computing the reservoir is in a way a random feature space.
  
  To sum up, the echo state condition ensures that the signal becomes linearly separable.
  
  > **Comments:**
  > **username_0**: Can you state this intuition in the form of a theorem? It is not clear to me how you get linear separability just from echo state (or even what "linearly separable" really means in the context of sequence-to-sequence transformations).  
  > **username_1**: In this articel (http://www.scholarpedia.org/article/Echo_state_network) Jaeger writes: "ESP states that the reservoir will asymptotically wash out any information from initial conditions." This "fading memory property" was formalized by Boyd et al (Fading Memory and the Problem of Approximating Nonlinear Operators with Volterra Series, 1985): for all epsilon there exists delta>0 and T >0 such that |F(v)(t=0)-F(u)(t=0)| < epsilon with |u(t) - v(t)| < delta for all t \in [-T,0].  
  > **username_1**: where F is some time-invariant, causal Operator (the reservoir in our case).  
  
  ---
  
  