Answers.Body: <p>You need to scale the input. Neural networks work best with a limited input domain, and train badly when it is exceeded.</p>
  
  <p>For statistical data, you would typically scale your input to have mean 0, standard deviation 1.</p>
  
  <p>Here, you will be better off fitting the input to roughly -1 to 1. </p>
  
  <p>Up to you where you scale the values, but usually this is done outside of the NN code. So I would do something like:</p>
  
  <pre><code>nn_input = (t - 15)/15
  </code></pre>
  
  <p>And then use <code>nn_input</code> in the training and evaluation loops. As you are putting these directly into a sorted array for plotting, you won't need to do any further re-mapping back or maintain a conversion function. However, in the more common case of arbitrary inputs, you would need to store the conversion factors somewhere (in this case just hardcoded as a function perhaps) in order to make use of the trained NN.</p>
  
  <p>Another thing that may help is shuffling your input/output data pairs during training to remove correlation between sequence of input pairs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-16T10:58:03.343Z
Answers.LastActivityDate: 2018-10-16T10:58:03.343Z
Answers.UserDisplayName: Neil Slater
Body: <p>I would like to get a simple example running in matlab that will use a neural net to learn an arbitrary function from input output data (basically model identification) and then be able to approximate that function from just the input data. As means of training this net I have implemented a simple back propagation algorithm in matlab but I was not able to get anywhere close to satisfactory results. I would like to know what I may be doing wrong and also what approach I may use instead.</p>
  
  <p>The goal is to have the network represent an identified function f(x) which takes a series x as input and outputs the learned mapping from x -> y. </p>
  
  <p>Here is the GNU octave code I have so far:</p>
  
  <pre><code>pkg load control signal
  
  function r = sigmoid(z)
      r = 1 ./ (1 + exp(-z));
  end
  
  function r = linear(z)
      r = z;
  end 
  
  function r = grad_sigmoid(z)
      r = sigmoid(z) .* (1 - sigmoid(z));
  end 
  
  function r = grad_linear(z)
      r = 1;
  end 
  
  function r = grad_tanh(z)
      r = 1 - tanh(z) .^ 2;
  end
  
  function nn = nn_init(n_input, n_hidden1, n_hidden2, n_output)
      nn.W2 = (rand(n_input, n_hidden1) * 2 - 1)'
      nn.W3 = (rand(n_hidden1, n_hidden2) * 2 - 1)'
      nn.W4 = (rand(n_hidden2, n_output) * 2 - 1)'
      nn.lambda = 0.005;
  end
  
  function nn = nn_train(nn_in, state, action)
      nn = nn_in;
  
      [out, nn] = nn_eval(nn, state);
  
      d4 = (nn.a4 - action) .* grad_linear(nn.W4 * nn.a3); 
      d3 = (nn.W4' * d4) .* grad_tanh(nn.W3 * nn.a2);
      d2 = (nn.W3' * d3) .* grad_tanh(nn.W2 * nn.a1);
  
      nn.W4 -= nn.lambda * (d4 * nn.a3');
      nn.W3 -= nn.lambda * (d3 * nn.a2');
      nn.W2 -= nn.lambda * (d2 * nn.a1');
  end
  
  function [out,nn] = nn_eval(nn_in, state)
      nn = nn_in;
  
      nn.z1 = state;
      nn.a1 = nn.z1;
  
      nn.a2 = tanh(nn.W2 * nn.a1);
      nn.a3 = tanh(nn.W3 * nn.a2);
      nn.a4 = linear(nn.W4 * nn.a3);
  
      out = nn.a4;
  end
  
  nn = nn_init(1, 100, 100, 1);
  t = 1:0.1:3.14*10;
  input = t;
  output = sin(input);
  learned = zeros(1, length(output));
  
  for j = 1:500
      for i = 1:length(input)
          nn = nn_train(nn, [input(i)], [output(i)]); 
      end
      j
  end
  
  for i = 1:length(input)
      learned(i) = nn_eval(nn, [input(i)]);    
  end
  
  plot(t, output, 'g', t, learned, 'b');
  
  pause
  </code></pre>
  
  <p>Here is the result:
  <a href="https://i.sstatic.net/IK8Gn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IK8Gn.png" alt="enter image description here"></a></p>
  
  <p>The result is not even close to where I want it to be. Has it got something to do with my implementation of back propagation? </p>
  
  <p>What changes do I need to do to the code to get a better approximation going? </p>
  
Tags: <feedforward-neural-networks>
Title: Learning an arbitrary function using a feedforward net
UserDisplayName: Martin
ThreadText: # Learning an arbitrary function using a feedforward net
  
  (Asked by: username_0 on 2018-10-16)
  
  I would like to get a simple example running in matlab that will use a neural net to learn an arbitrary function from input output data (basically model identification) and then be able to approximate that function from just the input data. As means of training this net I have implemented a simple back propagation algorithm in matlab but I was not able to get anywhere close to satisfactory results. I would like to know what I may be doing wrong and also what approach I may use instead.
  The goal is to have the network represent an identified function f(x) which takes a series x as input and outputs the learned mapping from x -> y. 
  Here is the GNU octave code I have so far:
  pkg load control signal
  
  function r = sigmoid(z)
      r = 1 ./ (1 + exp(-z));
  end
  
  function r = linear(z)
      r = z;
  end 
  
  function r = grad_sigmoid(z)
      r = sigmoid(z) .* (1 - sigmoid(z));
  end 
  
  function r = grad_linear(z)
      r = 1;
  end 
  
  function r = grad_tanh(z)
      r = 1 - tanh(z) .^ 2;
  end
  
  function nn = nn_init(n_input, n_hidden1, n_hidden2, n_output)
      nn.W2 = (rand(n_input, n_hidden1) * 2 - 1)'
      nn.W3 = (rand(n_hidden1, n_hidden2) * 2 - 1)'
      nn.W4 = (rand(n_hidden2, n_output) * 2 - 1)'
      nn.lambda = 0.005;
  end
  
  function nn = nn_train(nn_in, state, action)
      nn = nn_in;
  
      [out, nn] = nn_eval(nn, state);
  
      d4 = (nn.a4 - action) .* grad_linear(nn.W4 * nn.a3); 
      d3 = (nn.W4' * d4) .* grad_tanh(nn.W3 * nn.a2);
      d2 = (nn.W3' * d3) .* grad_tanh(nn.W2 * nn.a1);
  
      nn.W4 -= nn.lambda * (d4 * nn.a3');
      nn.W3 -= nn.lambda * (d3 * nn.a2');
      nn.W2 -= nn.lambda * (d2 * nn.a1');
  end
  
  function [out,nn] = nn_eval(nn_in, state)
      nn = nn_in;
  
      nn.z1 = state;
      nn.a1 = nn.z1;
  
      nn.a2 = tanh(nn.W2 * nn.a1);
      nn.a3 = tanh(nn.W3 * nn.a2);
      nn.a4 = linear(nn.W4 * nn.a3);
  
      out = nn.a4;
  end
  
  nn = nn_init(1, 100, 100, 1);
  t = 1:0.1:3.14*10;
  input = t;
  output = sin(input);
  learned = zeros(1, length(output));
  
  for j = 1:500
      for i = 1:length(input)
          nn = nn_train(nn, [input(i)], [output(i)]); 
      end
      j
  end
  
  for i = 1:length(input)
      learned(i) = nn_eval(nn, [input(i)]);    
  end
  
  plot(t, output, 'g', t, learned, 'b');
  
  pause
  
  Here is the result:
  
  The result is not even close to where I want it to be. Has it got something to do with my implementation of back propagation? 
  What changes do I need to do to the code to get a better approximation going?
  
  **Tags:** feedforward-neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-10-16. Score: 1 | Accepted answer
  
  You need to scale the input. Neural networks work best with a limited input domain, and train badly when it is exceeded.
  For statistical data, you would typically scale your input to have mean 0, standard deviation 1.
  Here, you will be better off fitting the input to roughly -1 to 1. 
  Up to you where you scale the values, but usually this is done outside of the NN code. So I would do something like:
  nn_input = (t - 15)/15
  
  And then use nn_input in the training and evaluation loops. As you are putting these directly into a sorted array for plotting, you won't need to do any further re-mapping back or maintain a conversion function. However, in the more common case of arbitrary inputs, you would need to store the conversion factors somewhere (in this case just hardcoded as a function perhaps) in order to make use of the trained NN.
  Another thing that may help is shuffling your input/output data pairs during training to remove correlation between sequence of input pairs.
  
  
  ---
  
  