Answers.Body: <p>This is because of <strong>Vanishing Gradient Problem</strong></p>
  
  <p>What is Vanishing Gradient Problem ?</p>
  
  <p>when we do Back-propagation i.e moving backward in the Network and calculating gradients of loss(Error) with respect to the weights , the gradients tends to get smaller and smaller as we keep on moving backward in the Network. This means that the neurons in the Earlier layers learn very slowly as compared to the neurons in the later layers in the Hierarchy. The Earlier layers in the network are slowest to train.</p>
  
  <p><strong>Reason</strong></p>
  
  <p>Sigmoid function, squishes a large input space into a small input space between 0 and 1. Therefore a large change in the input of the sigmoid function will cause a small change in the output. Hence, the derivative becomes small. <a href="https://i.sstatic.net/vKtiQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vKtiQ.png" alt="Sigmoid And its Derivative Function"></a></p>
  
  <p><strong>Solution:</strong></p>
  
  <p>Use Activation function as ReLu</p>
  
  <p><a href="https://i.sstatic.net/dAAFA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dAAFA.png" alt="ReLu"></a></p>
  
  <p>Reference:</p>
  
  <p><a href="https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484" rel="nofollow noreferrer">Vanishing Gradient Solution</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-24T01:31:45.683Z
Answers.Text: How do you know it's because of the vanishing gradient problem? OP's neural network is quite small, so, even though the OP using the sigmoid, I doubt that their problem is due to the vanishing gradient. In fact, the OP gets reasonable numbers (i.e. not too small), they are just wrong. I encourage you to have a look at their code first, before saying it's the vanishing gradient problem.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-24T01:33:28.083Z
Answers.Text: Sure i will look into it @nbro
Answers.UserDisplayName: DRV
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-24T01:26:40.380Z
Answers.LastActivityDate: 2020-03-24T01:26:40.380Z
Answers.UserDisplayName: DRV
Body: <p>I have this simple neural network in Python which I'm trying to use to aproximation tanh function. As inputs I have x - inputs to the function, and as outputs I want tanh(x) = y. I'm using sigmoid function also as an activation function of this neural network. </p>
  
  <pre><code>import numpy
  # scipy.special for the sigmoid function expit()
  import scipy.special
  # library for plotting arrays
  import matplotlib.pyplot
  # ensure the plots are inside this notebook, not an external window
  %matplotlib inline
  
  # neural network class definition
  class neuralNetwork:
  
  
      # initialise the neural network
      def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
          # set number of nodes in each input, hidden, output layer
          self.inodes = inputnodes
          self.hnodes = hiddennodes
          self.onodes = outputnodes
  
          # link weight matrices, wih and who
          # weights inside the arrays are w_i_j, where link is from node i to node j in the next layer
          # w11 w21
          # w12 w22 etc 
          self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
          self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))
  
          # learning rate
          self.lr = learningrate
  
          # activation function is the sigmoid function
          self.activation_function = lambda x: scipy.special.expit(x)  
  
          pass
  
  
      # train the neural network
      def train(self, inputs_list, targets_list):
          # convert inputs list to 2d array
          inputs = numpy.array(inputs_list, ndmin=2).T
          targets = numpy.array(targets_list, ndmin=2).T
  
          # calculate signals into hidden layer
          hidden_inputs = numpy.dot(self.wih, inputs)
          # calculate the signals emerging from hidden layer
          hidden_outputs = self.activation_function(hidden_inputs)
  
          # calculate signals into final output layer
          final_inputs = numpy.dot(self.who, hidden_outputs)
          # calculate the signals emerging from final output layer
          final_outputs = self.activation_function(final_inputs)
  
          # output layer error is the (target - actual)
          output_errors = targets - final_outputs
          # hidden layer error is the output_errors, split by weights, recombined at hidden nodes
          hidden_errors = numpy.dot(self.who.T, output_errors) 
  
          # BACKPROPAGATION &amp; gradient descent part, i.e updating weights first between hidden
          # layer and output layer, 
          # update the weights for the links between the hidden and output layers
          self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)), numpy.transpose(hidden_outputs))
  
          # update the weights for the links between the input and hidden layers, second part of backpropagation.
          self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)), numpy.transpose(inputs))
          pass
  
  
      # query the neural network
      def query(self, inputs_list):
          # convert inputs list to 2d array
          inputs = numpy.array(inputs_list, ndmin=2).T
  
          # calculate signals into hidden layer
          hidden_inputs = numpy.dot(self.wih, inputs)
          # calculate the signals emerging from hidden layer
          hidden_outputs = self.activation_function(hidden_inputs)
  
          # calculate signals into final output layer
          final_inputs = numpy.dot(self.who, hidden_outputs)
          # calculate the signals emerging from final output layer
          final_outputs = self.activation_function(final_inputs)
  
          return final_outputs
  </code></pre>
  
  <p>Now I try to query this network, This network has three input nodes one for each x, one node for each input. This network also has 3 output nodes, so It would classify the inputs to given outputs. Where outputs are y, y = tanh(x) function. </p>
  
  <pre><code># number of input, hidden and output nodes
  input_nodes = 3
  hidden_nodes = 8
  output_nodes = 3
  learning_rate = 0.1
  
  # create instance of neural network
  n = neuralNetwork(input_nodes,hidden_nodes,output_nodes, learning_rate)
  
  realInputs = []
  realInputs.append(1)
  realInputs.append(2)
  realInputs.append(3)
  
  # for x in (-3, 3):
  #     realInputs.append(x)
  #     pass
  
  expectedOutputs = []
  expectedOutputs.append(numpy.tanh(1));
  expectedOutputs.append(numpy.tanh(2));
  expectedOutputs.append(numpy.tanh(3));
  
  for y in expectedOutputs:
      print(y)
      pass
  
  training_data_list = []
  
  # epochs is the number of times the training data set is used for training
  epochs = 200
  
  for e in range(epochs):
      # go through all records in the training data set
      for record in training_data_list:
          # scale and shift the inputs
          inputs = realInputs
          targets = expectedOutputs
          n.train(inputs, targets)
          pass
      pass
  
  n.query(realInputs)
  </code></pre>
  
  <p>Outputs: desired vs ones from network with same data as training data:</p>
  
  <pre><code>0.7615941559557649
  0.9640275800758169
  0.9950547536867305
  
  
  array([[-0.21907413],
         [-0.6424568 ],
         [-0.25772344]])
  </code></pre>
  
  <p>My results are completely wrong. I'm a beginner with neural networks so I wanted to build neural network without frameworks like tensor flow... Could someone help me? Thank you. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-24T00:19:50.207Z
Comments.Text: Hi and welcome to AI SE! It may be a good idea if you describe a little bit your code. For example, 1. Where are you implementing the back-propagation algorithm? 2. Where are you implementing the gradient descent step? 3. Why do you have 3 input nodes? 4. Why do you have 3 output nodes?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-24T10:35:29.263Z
Comments.Text: @nbro Updated, added further description.
Comments.UserDisplayName: Ver
Tags: <neural-networks><backpropagation>
Title: Simple three layer neural network with backpropagation is not approximating tanh function
UserDisplayName: Ver
ThreadText: # Simple three layer neural network with backpropagation is not approximating tanh function
  
  (Asked by: username_0 on 2020-03-23)
  
  I have this simple neural network in Python which I'm trying to use to aproximation tanh function. As inputs I have x - inputs to the function, and as outputs I want tanh(x) = y. I'm using sigmoid function also as an activation function of this neural network.
  
  ```
  import numpy
  # scipy.special for the sigmoid function expit()
  import scipy.special
  # library for plotting arrays
  import matplotlib.pyplot
  # ensure the plots are inside this notebook, not an external window
  %matplotlib inline
  
  # neural network class definition
  class neuralNetwork:
  
      # initialise the neural network
      def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
          # set number of nodes in each input, hidden, output layer
          self.inodes = inputnodes
          self.hnodes = hiddennodes
          self.onodes = outputnodes
  
          # link weight matrices, wih and who
          # weights inside the arrays are w_i_j, where link is from node i to node j in the next layer
          # w11 w21
          # w12 w22 etc 
          self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
          self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))
  
          # learning rate
          self.lr = learningrate
  
          # activation function is the sigmoid function
          self.activation_function = lambda x: scipy.special.expit(x)  
  
          pass
  
      # train the neural network
      def train(self, inputs_list, targets_list):
          # convert inputs list to 2d array
          inputs = numpy.array(inputs_list, ndmin=2).T
          targets = numpy.array(targets_list, ndmin=2).T
  
          # calculate signals into hidden layer
          hidden_inputs = numpy.dot(self.wih, inputs)
          # calculate the signals emerging from hidden layer
          hidden_outputs = self.activation_function(hidden_inputs)
  
          # calculate signals into final output layer
          final_inputs = numpy.dot(self.who, hidden_outputs)
          # calculate the signals emerging from final output layer
          final_outputs = self.activation_function(final_inputs)
  
          # output layer error is the (target - actual)
          output_errors = targets - final_outputs
          # hidden layer error is the output_errors, split by weights, recombined at hidden nodes
          hidden_errors = numpy.dot(self.who.T, output_errors) 
  
          # BACKPROPAGATION & gradient descent part, i.e updating weights first between hidden
          # layer and output layer, 
          # update the weights for the links between the hidden and output layers
          self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)), numpy.transpose(hidden_outputs))
  
          # update the weights for the links between the input and hidden layers, second part of backpropagation.
          self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)), numpy.transpose(inputs))
          pass
  
      # query the neural network
      def query(self, inputs_list):
          # convert inputs list to 2d array
          inputs = numpy.array(inputs_list, ndmin=2).T
  
          # calculate signals into hidden layer
          hidden_inputs = numpy.dot(self.wih, inputs)
          # calculate the signals emerging from hidden layer
          hidden_outputs = self.activation_function(hidden_inputs)
  
          # calculate signals into final output layer
          final_inputs = numpy.dot(self.who, hidden_outputs)
          # calculate the signals emerging from final output layer
          final_outputs = self.activation_function(final_inputs)
  
          return final_outputs
  
  ```
  
  Now I try to query this network, This network has three input nodes one for each x, one node for each input. This network also has 3 output nodes, so It would classify the inputs to given outputs. Where outputs are y, y = tanh(x) function.
  
  ```
  # number of input, hidden and output nodes
  input_nodes = 3
  hidden_nodes = 8
  output_nodes = 3
  learning_rate = 0.1
  
  # create instance of neural network
  n = neuralNetwork(input_nodes,hidden_nodes,output_nodes, learning_rate)
  
  realInputs = []
  realInputs.append(1)
  realInputs.append(2)
  realInputs.append(3)
  
  # for x in (-3, 3):
  #     realInputs.append(x)
  #     pass
  
  expectedOutputs = []
  expectedOutputs.append(numpy.tanh(1));
  expectedOutputs.append(numpy.tanh(2));
  expectedOutputs.append(numpy.tanh(3));
  
  for y in expectedOutputs:
      print(y)
      pass
  
  training_data_list = []
  
  # epochs is the number of times the training data set is used for training
  epochs = 200
  
  for e in range(epochs):
      # go through all records in the training data set
      for record in training_data_list:
          # scale and shift the inputs
          inputs = realInputs
          targets = expectedOutputs
          n.train(inputs, targets)
          pass
      pass
  
  n.query(realInputs)
  
  ```
  
  Outputs: desired vs ones from network with same data as training data:
  
  ```
  0.7615941559557649
  0.9640275800758169
  0.9950547536867305
  
  array([[-0.21907413],
         [-0.6424568 ],
         [-0.25772344]])
  
  ```
  
  My results are completely wrong. I'm a beginner with neural networks so I wanted to build neural network without frameworks like tensor flow... Could someone help me? Thank you.
  
  **Tags:** neural-networks, backpropagation
  
  > **Comments:**
  > **username_1**: Hi and welcome to AI SE! It may be a good idea if you describe a little bit your code. For example, 1. Where are you implementing the back-propagation algorithm? 2. Where are you implementing the gradient descent step? 3. Why do you have 3 input nodes? 4. Why do you have 3 output nodes?  
  > **username_0**: @username_1 Updated, added further description.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-03-24. Score: -1
  
  This is because of **Vanishing Gradient Problem**
  
  What is Vanishing Gradient Problem ?
  
  when we do Back-propagation i.e moving backward in the Network and calculating gradients of loss(Error) with respect to the weights , the gradients tends to get smaller and smaller as we keep on moving backward in the Network. This means that the neurons in the Earlier layers learn very slowly as compared to the neurons in the later layers in the Hierarchy. The Earlier layers in the network are slowest to train.
  
  **Reason**
  
  Sigmoid function, squishes a large input space into a small input space between 0 and 1. Therefore a large change in the input of the sigmoid function will cause a small change in the output. Hence, the derivative becomes small. [![Sigmoid And its Derivative Function](https://i.sstatic.net/vKtiQ.png)](https://i.sstatic.net/vKtiQ.png)
  
  **Solution:**
  
  Use Activation function as ReLu
  
  [![ReLu](https://i.sstatic.net/dAAFA.png)](https://i.sstatic.net/dAAFA.png)
  
  Reference:
  
  [Vanishing Gradient Solution](https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484)
  
  > **Comments:**
  > **username_1**: How do you know it's because of the vanishing gradient problem? OP's neural network is quite small, so, even though the OP using the sigmoid, I doubt that their problem is due to the vanishing gradient. In fact, the OP gets reasonable numbers (i.e. not too small), they are just wrong. I encourage you to have a look at their code first, before saying it's the vanishing gradient problem.  
  > **username_2**: Sure i will look into it @username_1  
  
  ---
  
  