Answers.Body: <h1>Local minima.</h1>
  <p>You have the exact same issue of this <a href="https://ai.stackexchange.com/questions/5904/why-does-a-one-layer-hidden-network-get-more-robust-to-poor-initialization-with/">question</a>. If you randomize your initial weights, you'll see sometimes you get the correct results, and others you won't. It's because when the weights are initialized with a certain range of values, they will converge to a local minima which you cannot escape with a low learning rate.</p>
  <p>A simple solution is to increase the size of your hidden layer, which will make the network more robust to such issues.</p>
  <blockquote>
  <p>When you have only 2 dimensions, a local minima exists. When you have more dimensions, this minima gets harder and harder to reach, as its likelihood decreases. Intuitively, you have a lot more dimensions through which you can improve than if you only had 2 dimensions.</p>
  <p>The problem still exists, even with 1000 neurons you could find a specific set of weights which was a local minimum. However, it just becomes so much less likely.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-26T12:46:43.530Z
Answers.Text: XD So that is why 2 layers failed. I have some changes to make.
Answers.UserDisplayName: FreezePhoenix
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-27T08:15:35.643Z
Answers.Text: I see. So is it just a complete luck that if I ran Daniel's code, I get correct answer 9/10 times and if I run mine, I get the correct answer only 2/10 times?
Answers.UserDisplayName: Gabriele
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-27T08:18:47.063Z
Answers.Text: It has to do with initialization, most likely. I havent seen his code, but i assume he uses some initialization method different than yours
Answers.UserDisplayName: BlueMoon93
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-26T12:34:55.227Z
Answers.LastActivityDate: 2018-04-26T12:34:55.227Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: BlueMoon93
Body: <p>I was following <a href="http://natureofcode.com/" rel="nofollow noreferrer">Daniel Shiffman's</a> tutorials on how to write your own neural network from scratch. I specifically looked into his <a href="https://www.youtube.com/watch?v=MPmLWsHzPlU" rel="nofollow noreferrer">videos</a> and the code he provided in <a href="https://github.com/CodingTrain/website/blob/master/Courses/natureofcode/10.18-toy_neural_network/lib/nn.js" rel="nofollow noreferrer">here</a>. I rewrote his code in Python, however, 3 out of 4 of my outputs are the same. The neural network has two input nodes, one hidden layer with two nodes and one output node. Can anyone help me to find my mistake? Here is my <a href="https://pastebin.com/saJxedDT" rel="nofollow noreferrer">full code</a>. </p>
  
  <pre><code>import random
  
  nn = NeuralNetwork(2,2,1)
  inputs  = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])
  targets = np.array([[0], [1], [1], [0]])
  zipped = zip(inputs, targets)
  list_zipped = list(zipped)
  
  for _ in range(9000):
      x, y = random.choice(list_zipped)
      nn.train(x, y)
  
  output = [nn.feedforward(i) for i in inputs]
  
  for i in output:
     print("Output ", i)
  
  #Output  [ 0.1229546]  when it should be around 0
  #Output  [ 0.6519492]  ~1
  #Output  [ 0.65180228] ~1
  #Output  [ 0.66269853] ~0
  </code></pre>
  
  <p><strong>EDIT_1:</strong> I tried debugging my code by choosing all weights and bias' values to 0.5. I did this in both my code and Daniel's. This obviously ended up showing me <em>all</em> outputs with the same value. </p>
  
  <p>After that I increased my weights and bias' values variety from [0 , 1) to [-1, 1). By running this a few times, I would <strong>sometimes</strong> get the correct output: </p>
  
  <pre><code>[ 0.93749991] # should be ~1
  [ 0.93314793] # ~1 
  [ 0.07001175] # ~0
  [ 0.06576194] # ~0
  </code></pre>
  
  <p>If I ran nn.train() 100 000 times, I get the correct output 2/3 times. 
  Is this the issue of gradient descent, where it converges to the local minima?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-26T12:49:41.853Z
Comments.Text: You mean "two input nodes, 1 hidden layer with 2 neurons, and one output node" instead of "two input layers, 2 hidden layers and one output layers" right?
Comments.UserDisplayName: BlueMoon93
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-27T08:11:18.667Z
Comments.Text: Yes, I will fix my question.
Comments.UserDisplayName: Gabriele
Tags: <neural-networks><python>
Title: Neural network returns similar output
UserDisplayName: Gabriele
ThreadText: # Neural network returns similar output
  
  (Asked by: username_0 on 2018-04-25)
  
  I was following Daniel Shiffman's tutorials on how to write your own neural network from scratch. I specifically looked into his videos and the code he provided in here. I rewrote his code in Python, however, 3 out of 4 of my outputs are the same. The neural network has two input nodes, one hidden layer with two nodes and one output node. Can anyone help me to find my mistake? Here is my full code. 
  import random
  
  nn = NeuralNetwork(2,2,1)
  inputs  = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])
  targets = np.array([[0], [1], [1], [0]])
  zipped = zip(inputs, targets)
  list_zipped = list(zipped)
  
  for _ in range(9000):
      x, y = random.choice(list_zipped)
      nn.train(x, y)
  
  output = [nn.feedforward(i) for i in inputs]
  
  for i in output:
     print("Output ", i)
  
  #Output  [ 0.1229546]  when it should be around 0
  #Output  [ 0.6519492]  ~1
  #Output  [ 0.65180228] ~1
  #Output  [ 0.66269853] ~0
  
  EDIT_1: I tried debugging my code by choosing all weights and bias' values to 0.5. I did this in both my code and Daniel's. This obviously ended up showing me all outputs with the same value. 
  After that I increased my weights and bias' values variety from [0 , 1) to [-1, 1). By running this a few times, I would sometimes get the correct output: 
  [ 0.93749991] # should be ~1
  [ 0.93314793] # ~1 
  [ 0.07001175] # ~0
  [ 0.06576194] # ~0
  
  If I ran nn.train() 100 000 times, I get the correct output 2/3 times. 
  Is this the issue of gradient descent, where it converges to the local minima?
  
  **Tags:** neural-networks, python
  > **Comments:**
  > **username_1**: You mean "two input nodes, 1 hidden layer with 2 neurons, and one output node" instead of "two input layers, 2 hidden layers and one output layers" right?  
  > **username_0**: Yes, I will fix my question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-04-26. Score: 2 | Accepted answer
  
  # Local minima.
  
  You have the exact same issue of this [question](https://ai.stackexchange.com/questions/5904/why-does-a-one-layer-hidden-network-get-more-robust-to-poor-initialization-with/). If you randomize your initial weights, you'll see sometimes you get the correct results, and others you won't. It's because when the weights are initialized with a certain range of values, they will converge to a local minima which you cannot escape with a low learning rate.
  
  A simple solution is to increase the size of your hidden layer, which will make the network more robust to such issues.
  
  > When you have only 2 dimensions, a local minima exists. When you have more dimensions, this minima gets harder and harder to reach, as its likelihood decreases. Intuitively, you have a lot more dimensions through which you can improve than if you only had 2 dimensions.
  >
  > The problem still exists, even with 1000 neurons you could find a specific set of weights which was a local minimum. However, it just becomes so much less likely.
  
  > **Comments:**
  > **username_2**: XD So that is why 2 layers failed. I have some changes to make.  
  > **username_0**: I see. So is it just a complete luck that if I ran Daniel's code, I get correct answer 9/10 times and if I run mine, I get the correct answer only 2/10 times?  
  > **username_1**: It has to do with initialization, most likely. I havent seen his code, but i assume he uses some initialization method different than yours  
  
  ---
  
  