title: Echo State Network learning Mackey-Glass function, but how?
body: <p>I got this <a href="http://web.archive.org/web/20140806020438/http://minds.jacobs-university.de/mantas/code" rel="nofollow noreferrer">example of a minimal Echo State Network</a> (ESN) which I analyse while trying to understand <a href="http://www.scholarpedia.org/article/Echo_state_network" rel="nofollow noreferrer">Echo State Networks</a>. Unfortunately I have some problems understanding why this really works. It all breaks down to the questions:</p>
  <ul>
  <li>[ What defines | What is] the <em>echo state</em> of an ESN?</li>
  <li>What is it that makes an ESN so easy and fast learning of such complex nonlinear functions like the  Mackey-Glass function?</li>
  </ul>
  <p>First here is a little piece of code that shows the important part of initialization:</p>
  <pre><code>% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  %
  % Generate the ESN reservoir
  % 
  % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  
  rand('seed', 42);
  
  trainLen = 2000;
  testLen  = 2000;
  initLen  = 100;
  data     = load('MackeyGlass_t17.txt');
  
  %         Input neurons
  inSize  = 1; 
  %         Output neurons 
  outSize = 1;
  %         Reservoir size
  resSize = 1000;
  %         Leaking rate
  a       = 0.3; 
  %         Input weights
  Win     = ( rand(resSize, (inSize+1) ) - 0.5) .* 1;
  %         Reservoir weights
  W       = rand(resSize, resSize) - 0.5;
  </code></pre>
  <hr />
  <h1>Running the reservoir:</h1>
  <p>I understand that every single data-point of the input data set is propagated from the input neuron to the reservoir neurons. After a warm-up of size <code>initLen</code> the states are accepted and stored in matrix <code>X</code>. When this is done every single column of <code>X</code> represents a &quot;<em>vector of reservoir neuron activations</em>&quot;. And here comes the point where I am not sure if I got it right:</p>
  <p>The comment already says &quot;<em>collected states</em>&quot; or &quot;<em>design matrix</em>&quot; <code>X</code>. Am I getting this right, that all this does is storing the state of the whole network in the rows of matrix <code>X</code>?</p>
  <ul>
  <li>If we assume that <code>t</code> was just a time parameter then <code>X(:,t)</code> represents the network state of time <code>t </code>, isn't it?</li>
  </ul>
  <p>In my examples this would mean that there are 1.900 time slices which represent the whole network state of their corresponding timeframe (<code>X</code> therefore is a <code>1002x1900</code> matrix). Another question that occurs to me here is</p>
  <ul>
  <li>why is a <code>1</code> (I guess it is the bias) and the input value <code>u</code> appended to this vector: <code>X(:,t-initLen) = [1;u;x];</code></li>
  </ul>
  <p>So:</p>
  <pre><code>% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  % 
  % Run the reservoir with the data and collect X.
  % 
  % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  
  %       Allocated memory for the design (collected states) matrix
  X     = zeros((1+inSize) + resSize, trainLen - initLen);
  
  %       Vector of reservoir neuron activations (used for calculation)
  x     = zeros(resSize, 1);
  
  %       Update of the reservoir neuron activations
  xUpd  = zeros(resSize, 1);
  
  for t = 1:trainLen
      
      u    = data(t);
      
      xUpd = tanh( Win * [1;u] + W * x );    
      x    = (1-a) * x + a * xUpd;
      
      if ( t &gt; initLen )
          X(:,t-initLen) = [1;u;x];
      end
      
  end
  </code></pre>
  <hr />
  <h1>Training part:</h1>
  <p>The training part is also a little <em>magic</em> to me yet. I am familiar how linear regression works, so this is not the problem here.</p>
  <p>What I see is that this part just uses the hole state matrix <code>X</code> and performs a single linear regression step on the input data to generate the output weight vector <code>Wout</code> and that's it.</p>
  <p>So all that's been done so far - if I'm not mistaken - is initializing the output weights according to the state matri <code>X</code> which itself was generated using input data and randomly gernerated (input and reservoir) weights.</p>
  <pre><code>% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  % 
  % Train the output
  % 
  % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  
  %       Set the corresponding target matrix directly
  Yt    = data(initLen+2:trainLen+1)';
  
  %       Regularization coefficient
  reg   = 1e-8;  
  
  %       Get X transposed - needed twice therefore it is a little faster
  X_T   = X';
  
  %       Yt * pseudo_inverse(X); (linear regression task)
  Wout  = Yt * X_T * (X * X_T + reg * eye(1+inSize+resSize))^(-1);
  </code></pre>
  <hr />
  <h1>Running the ESN in a generative mode:</h1>
  <p>I can run this in two modes: generative or predictive. But well, this is the part where I just can say: &quot;<em>Well, .. it works.</em>&quot; not having the exact idea why it is.</p>
  <pre><code>% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  % 
  % Run the trained ESN in a generative mode. no need to initialize here, 
  % because x is initialized with training data and we continue from there.
  % 
  % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  
  Y = zeros(outSize,testLen);
  u = data(trainLen+1);
  
  for t = 1:testLen 
      
      xUpd   = tanh( Win*[1;u] + W*x );
      x      = (1-a)*x + a*xUpd;
      
      %        Generative mode:
      u      = Wout*[1;u;x];
  
      %      This would be a predictive mode:
      %u      = data(trainLen+t+1);
  
      Y(:,t) = u;
      
  end
  </code></pre>
  <p>It works pretty well as you can see (generative mode):</p>
  <p><img src="https://i.stack.imgur.com/Qdxxz.jpg" alt="enter image description here" /></p>
  <p>I know this is a quiet huge &quot;<em>question</em>&quot; if this can even be considered as one. I feel like I am understanding the single parts but what I'm missing is <strong>the big picture</strong> of this magic black box called Echo State Network.</p>