title: Neural network online training
body: <blockquote>
    <p>I want to implement a simple feed-forward neural network to approximate the function <strong><em>y=f(x)=ax^2</em></strong> where <strong><em>a</em></strong> is some constant and <strong><em>x</em></strong> is the input value. </p>
  </blockquote>
  
  <p>The NN has <strong>one input node</strong>, <strong>one hidden layer with 1-n nodes</strong>, and <strong>one output node</strong>. For example, I input the value 2.0 -> the NN produces 4.0, and again I input 3.0 -> the NN produces 9.0 or close to it and so on. </p>
  
  <p>If I understand "online-training," the training data is fed one by one - meaning I input the value 2.0 -> I iterate with the gradient decent 100 times, and then I pass the value 3.0, and I iterate another 100 times. </p>
  
  <p>However, when I try to do this with my experimental/learning NN - I input the value 2.0 -> the error gets very small -> the output is very close to 4.0.</p>
  
  <p>Now if I want to predict for the input 3.0 -> the NN produces 4.36 or something instead of 9.0. So the NN just learns the last training value. </p>
  
  <blockquote>
    <p>How can I use online-training to get a Neural Network that approximates the desired function for a range [-d, d]? What am I missing?</p>
  </blockquote>
  
  <p>The reason why I like online-training is that eventually I want to input a time series - and map that series to the desired function. This is besides the point but in case someone was wondering. </p>
  
  <p>Any advise would be greatly appreciated.      </p>
  
  <p>More info - I am activating the hidden layer with the Sigmoid function and the output layer with the linear one. </p>