Answers.Body: <p><strong>Normalise your inputs.</strong></p>
  
  <p>Neural networks work poorly outside of relatively small numerical ranges on input. An ideal range is for each feature to be drawn from <span class="math-container">$\mathcal{N}(0,1)$</span> i.e. a Normal distribution with mean <span class="math-container">$0$</span> and standard deviation <span class="math-container">$1$</span>. In your case, divide both parts of <span class="math-container">$\mathbf{x}$</span> by <span class="math-container">$25$</span> and subtract <span class="math-container">$1$</span> would probably suffice.</p>
  
  <p>Your neural network architecture is completely overblown for the problem at hand. That may be because you were trying to force it to fit this data (and failing because of lack of normalisation). Try something more like: <code>hidden_units = [20,10]</code></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-23T08:27:16.227Z
Answers.Text: my inputs are now divided by 50 to be in range [0,1], but accuracy is still 83% (=5/6)
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-23T08:29:40.490Z
Answers.Text: any other changes that i should apply?
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-23T08:30:03.883Z
Answers.Text: @datdinhquoc: I'll take a deeper look. You probably need to play with other hyperparameters, such as learning rate, in order to get the classifier to split class 2 and 3 given just that one example. It may be an interesting toy example because of that, although typically you would not use a NN on real data with only 6 rows and 4 classes (one of which is not even in the data set)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-23T10:17:19.783Z
Answers.Text: @datdinhquoc: I can get 100% accuracy (in Keras) reliably using a simple network as I suggested, and around 500 epochs. It did take a few attempts with different hyperparameters. Probably one thing worth knowing is that `tanh` activations on hidden layers works better than `relu` for smaller networks. However, `relu` will work, you just need to make the NN slightly larger.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-23T10:24:03.613Z
Answers.Text: tks, i'll try switching the code to keras, may i have your 100%ac code? maybe in google colab?
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-23T17:14:53.250Z
Answers.Text: @datdinhquoc OK, here is a gist: https://gist.github.com/neilslater/6053e3729045e575f40f4d884f4d65dc - it is just hacky code. I replicated your data as train, cv and test - it is important to not do that for real data if you want to test whether your NN has generalised correctly. Instead you should split your dataset properly into separate datasets. If you want to know more about that. please research it or ask a new question on the site
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-25T08:07:38.893Z
Answers.Text: tks for the gist
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-23T07:41:43.830Z
Answers.LastActivityDate: 2019-08-23T08:23:10.400Z
Answers.LastEditDate: 2019-08-23T08:23:10.400Z
Answers.UserDisplayName: Neil Slater
Body: <p>The ready-to-use DNNClassifier in tf.estimator seems not able to fit these data:</p>
  
  <pre><code>X = [[1,2], [1,12], [1,17], [9,33], [48,49], [48,50]]
  Y = [ 1,     1,      1,      1,      2,       3     ]
  </code></pre>
  
  <p>I've tried with 4 layers but it's fitting to 83% (=5/6 sampes) only:</p>
  
  <pre><code>hidden_units = [2000,1000,500,100]
  n_classes    = 4   
  </code></pre>
  
  <p>The sample data above are supposed to be separated by 2 lines (right-click image to open in new tab):</p>
  
  <p><a href="https://i.sstatic.net/XT3OJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XT3OJ.png" alt="enter image description here"></a></p>
  
  <p>It seems stuck be cause of Y=2 and Y=3 are too close. How to change the DNNClassifier to fit to 100%?</p>
  
Tags: <deep-learning><classification><tensorflow><linear-regression><underfitting>
Title: TensorFlow estimator DNNClassifier fails to fit simple data
UserDisplayName: Dan D.
ThreadText: # TensorFlow estimator DNNClassifier fails to fit simple data
  
  (Asked by: username_0 on 2019-08-23)
  
  The ready-to-use DNNClassifier in tf.estimator seems not able to fit these data:
  X = [[1,2], [1,12], [1,17], [9,33], [48,49], [48,50]]
  Y = [ 1,     1,      1,      1,      2,       3     ]
  
  I've tried with 4 layers but it's fitting to 83% (=5/6 sampes) only:
  hidden_units = [2000,1000,500,100]
  n_classes    = 4   
  
  The sample data above are supposed to be separated by 2 lines (right-click image to open in new tab):
  
  It seems stuck be cause of Y=2 and Y=3 are too close. How to change the DNNClassifier to fit to 100%?
  
  **Tags:** deep-learning, classification, tensorflow, linear-regression, underfitting
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-23. Score: 2 | Accepted answer
  
  **Normalise your inputs.**
  
  Neural networks work poorly outside of relatively small numerical ranges on input. An ideal range is for each feature to be drawn from $\mathcal{N}(0,1)$ i.e. a Normal distribution with mean $0$ and standard deviation $1$. In your case, divide both parts of $\mathbf{x}$ by $25$ and subtract $1$ would probably suffice.
  
  Your neural network architecture is completely overblown for the problem at hand. That may be because you were trying to force it to fit this data (and failing because of lack of normalisation). Try something more like: `hidden_units = [20,10]`
  
  > **Comments:**
  > **username_0**: my inputs are now divided by 50 to be in range [0,1], but accuracy is still 83% (=5/6)  
  > **username_0**: any other changes that i should apply?  
  > **username_1**: @datdinhquoc: I'll take a deeper look. You probably need to play with other hyperparameters, such as learning rate, in order to get the classifier to split class 2 and 3 given just that one example. It may be an interesting toy example because of that, although typically you would not use a NN on real data with only 6 rows and 4 classes (one of which is not even in the data set)  
  > **username_1**: @datdinhquoc: I can get 100% accuracy (in Keras) reliably using a simple network as I suggested, and around 500 epochs. It did take a few attempts with different hyperparameters. Probably one thing worth knowing is that `tanh` activations on hidden layers works better than `relu` for smaller networks. However, `relu` will work, you just need to make the NN slightly larger.  
  > **username_0**: tks, i'll try switching the code to keras, may i have your 100%ac code? maybe in google colab?  
  > **username_1**: @datdinhquoc OK, here is a gist: https://gist.github.com/neilslater/6053e3729045e575f40f4d884f4d65dc - it is just hacky code. I replicated your data as train, cv and test - it is important to not do that for real data if you want to test whether your NN has generalised correctly. Instead you should split your dataset properly into separate datasets. If you want to know more about that. please research it or ask a new question on the site  
  > **username_0**: tks for the gist  
  
  ---
  
  