Answers.Body: <p>You should try bigger learning rate / more epochs.</p>
  
  <p>What you see is basically output of randomly initialized weights. Default learning rate for SGD is 0.001, it's definitely not enough for weights to change quick enough with single batch epochs and 9 examples.</p>
  
  <p>I never used TF.js, but in keras (that's look very similar) you can set it like this</p>
  
  <pre><code>sgd = keras.optimizers.SGD(lr=1)
  model.compile(sgd, 'mse')
  
  model.fit(xs, ys, epochs=200)
  </code></pre>
  
  <p>Another approach would be to use adaptive learning rate optimizer, for example adadelta. It still requires quite a lot of epochs, but much less, then SGD with standard learning rate and you don't need to set LR manually.</p>
  
  <pre><code>model.compile('adadelta', 'mse')
  model.fit(xs, ys, epochs=2000)
  </code></pre>
  
  <p><a href="http://ruder.io/optimizing-gradient-descent/" rel="nofollow noreferrer">Here</a> you can read how different optimizers work and converge </p>
  
  <p><a href="https://i.sstatic.net/6HZFE.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6HZFE.gif" alt="Optimizers convergency"></a></p>
  
  <p>Edit:
  Another thing with tf.js (and javascript code in general) is it doesn't guarantee order of operations. So, in your case it prints evaluation <em>before</em> it finishes training.</p>
  
  <p>This code should perform fine. Take note that we evaluate model in <code>then</code> clause.</p>
  
  <pre><code>const model = tf.sequential();
  model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
  model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
  model.compile({loss: 'meanSquaredError', optimizer: 'adadelta'});
  const xs = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9]);
  const ys = tf.tensor1d([0, 0, 0, 0, 0, 1, 1, 1, 1]);
  model.fit(xs, ys, {epochs: 1000}).then(h =&gt; {
     console.log("Loss: " + h.history.loss[0]);
     model.predict(xs).print();
  });
  </code></pre>
  
  <p>You can also use async/await approach, for example as it described <a href="https://stackoverflow.com/a/50705155/1173441">in this answer</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-13T10:59:21.017Z
Answers.Text: I tried experimenting with different amounts of epochs, higher learning rates and Adadelta optimizer, but all of it to no avail. I noticed that usually, the amount of not detected trends is approximately also the amount of correctly detected trends for every setting of the network (by setting I mean a certain optimizer and amount of epochs, and for every optimizer and amount of epochs I repeated the training process over and over again to see how the network would perform). Would this suggest that its performance is dependent on its starting position (the randomly chosen weight and bias)?
Answers.UserDisplayName: DaddyMike
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-13T14:00:16.990Z
Answers.Text: Performance indeed depends on network seed, but it would only influence convergency rate for that simple example, because there is no local minimum, only global one. I made keras implementation, it got 100% accuracy for each of 10 runs I did, you can try yourself https://gist.github.com/mephody-bro/49700ee69c78eabd5af1d6a818fb7ef6.
Answers.UserDisplayName: Kirill Fedyanin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-13T14:36:50.893Z
Answers.Text: (see edit) Same network, same functions, same amount of epochs... If javascript reads the tensors correctly and the default js batchsize is just the whole data set then I have literally no idea why we are getting different results.
Answers.UserDisplayName: DaddyMike
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-14T15:47:59.880Z
Answers.Text: I've updated answer; It seems you also need to force tf.js execute operations in right order.
Answers.UserDisplayName: Kirill Fedyanin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T16:08:06.317Z
Answers.LastActivityDate: 2019-08-14T15:50:51.813Z
Answers.LastEditDate: 2019-08-14T15:50:51.813Z
Answers.UserDisplayName: Kirill Fedyanin
Body: <p>I have created a <code>Tf.Sequential</code> model which outputs 1 for numbers bigger then 5 and 0 otherwise:</p>
  
  
  
  <pre><code>const model = tf.sequential();
  model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
  model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
  model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
  const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
  const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
  model.fit(xs, ys);
  model.predict(xs).print();
  </code></pre>
  
  <p>With 5 hidden neurons, not even the right trend is detected. Sometimes all the number are too low, or the outputs decrease even if the inputs increase or the outputs are too high.</p>
  
  <p>I later thought that the best way to do this is to have 2 neurons, where 1 is for the input and the other applies a sigmoid function to the input. The weight and bias should easily be adjusted to make the ANN work.</p>
  
  
  
  <pre><code>const model = tf.sequential();
  model.add(tf.layers.dense({ units: 1, activation: 'sigmoid', inputShape: [1]}));
  model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
  const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
  const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
  model.fit(xs, ys);
  model.predict(xs).print();
  </code></pre>
  
  <p>Sometimes, this ANN does detect the right trend (the higher the input, the higher the output), but still, the results are never correct and are usually simply too high, always providing an output too close to 1.</p>
  
  <p>How do I make my ANN work, and what have I done wrong?</p>
  
  <p><strong>Edit:</strong></p>
  
  <p>This is the code I'm using now, same problem as before:</p>
  
  
  
  <pre><code>const AdadeltaOptimizer = tf.train.adadelta();
  
  const model = tf.sequential();
  model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
  model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
  model.compile({loss: 'meanSquaredError', optimizer: AdadeltaOptimizer});
  const xs = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9]);
  const ys = tf.tensor1d([0, 0, 0, 0, 0, 1, 1, 1, 1]);
  model.fit(xs, ys, {
  epochs: 2000,
  });
  model.predict(xs).print();
  
  tf.losses.meanSquaredError(ys, model.predict(xs)).print();
  </code></pre>
  
Tags: <tensorflow>
Title: Why is this simple neural network not training?
UserDisplayName: DaddyMike
ThreadText: # Why is this simple neural network not training?
  
  (Asked by: username_0 on 2019-08-06)
  
  I have created a Tf.Sequential model which outputs 1 for numbers bigger then 5 and 0 otherwise:
  const model = tf.sequential();
  model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
  model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
  model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
  const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
  const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
  model.fit(xs, ys);
  model.predict(xs).print();
  
  With 5 hidden neurons, not even the right trend is detected. Sometimes all the number are too low, or the outputs decrease even if the inputs increase or the outputs are too high.
  I later thought that the best way to do this is to have 2 neurons, where 1 is for the input and the other applies a sigmoid function to the input. The weight and bias should easily be adjusted to make the ANN work.
  const model = tf.sequential();
  model.add(tf.layers.dense({ units: 1, activation: 'sigmoid', inputShape: [1]}));
  model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
  const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
  const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
  model.fit(xs, ys);
  model.predict(xs).print();
  
  Sometimes, this ANN does detect the right trend (the higher the input, the higher the output), but still, the results are never correct and are usually simply too high, always providing an output too close to 1.
  How do I make my ANN work, and what have I done wrong?
  Edit:
  This is the code I'm using now, same problem as before:
  const AdadeltaOptimizer = tf.train.adadelta();
  
  const model = tf.sequential();
  model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
  model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
  model.compile({loss: 'meanSquaredError', optimizer: AdadeltaOptimizer});
  const xs = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9]);
  const ys = tf.tensor1d([0, 0, 0, 0, 0, 1, 1, 1, 1]);
  model.fit(xs, ys, {
  epochs: 2000,
  });
  model.predict(xs).print();
  
  tf.losses.meanSquaredError(ys, model.predict(xs)).print();
  
  **Tags:** tensorflow
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-06. Score: 3 | Accepted answer
  
  You should try bigger learning rate / more epochs.
  What you see is basically output of randomly initialized weights. Default learning rate for SGD is 0.001, it's definitely not enough for weights to change quick enough with single batch epochs and 9 examples.
  I never used TF.js, but in keras (that's look very similar) you can set it like this
  sgd = keras.optimizers.SGD(lr=1)
  model.compile(sgd, 'mse')
  
  model.fit(xs, ys, epochs=200)
  
  Another approach would be to use adaptive learning rate optimizer, for example adadelta. It still requires quite a lot of epochs, but much less, then SGD with standard learning rate and you don't need to set LR manually.
  model.compile('adadelta', 'mse')
  model.fit(xs, ys, epochs=2000)
  
  Here you can read how different optimizers work and converge 
  
  Edit:
  Another thing with tf.js (and javascript code in general) is it doesn't guarantee order of operations. So, in your case it prints evaluation before it finishes training.
  This code should perform fine. Take note that we evaluate model in then clause.
  const model = tf.sequential();
  model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
  model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
  model.compile({loss: 'meanSquaredError', optimizer: 'adadelta'});
  const xs = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9]);
  const ys = tf.tensor1d([0, 0, 0, 0, 0, 1, 1, 1, 1]);
  model.fit(xs, ys, {epochs: 1000}).then(h => {
     console.log("Loss: " + h.history.loss[0]);
     model.predict(xs).print();
  });
  
  You can also use async/await approach, for example as it described in this answer
  
  > **Comments:**
  > **username_0**: I tried experimenting with different amounts of epochs, higher learning rates and Adadelta optimizer, but all of it to no avail. I noticed that usually, the amount of not detected trends is approximately also the amount of correctly detected trends for every setting of the network (by setting I mean a certain optimizer and amount of epochs, and for every optimizer and amount of epochs I repeated the training process over and over again to see how the network would perform). Would this suggest that its performance is dependent on its starting position (the randomly chosen weight and bias)?  
  > **username_1**: Performance indeed depends on network seed, but it would only influence convergency rate for that simple example, because there is no local minimum, only global one. I made keras implementation, it got 100% accuracy for each of 10 runs I did, you can try yourself https://gist.github.com/mephody-bro/49700ee69c78eabd5af1d6a818fb7ef6.  
  > **username_0**: (see edit) Same network, same functions, same amount of epochs... If javascript reads the tensors correctly and the default js batchsize is just the whole data set then I have literally no idea why we are getting different results.  
  > **username_1**: I've updated answer; It seems you also need to force tf.js execute operations in right order.  
  
  ---
  
  