Answers.Body: <p>It appears that you are adding the partial derivates (gradient) to make adjustments rather than subtracting them.  Remember that the gradient vector points &quot;upward.&quot;  You may find things improve dramatically if you subtract them (head downward) to perform gradient <em>descent</em>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-22T07:29:14.557Z
Answers.Text: Then you think my formula for the gradient descent is incorrect and I need to replace - with +? In that case, the problem persists
Answers.UserDisplayName: tail
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-22T14:06:41.463Z
Answers.Text: Your learning rate is also quite large in my opinion.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-22T14:16:10.870Z
Answers.Text: The problem persists for `l_rate = 0.000005` as well
Answers.UserDisplayName: tail
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-22T14:34:01.650Z
Answers.Text: I think there must be a bug somewhere in the code, but I cannot spot where yet
Answers.UserDisplayName: tail
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-21T22:30:39.843Z
Answers.LastActivityDate: 2022-09-21T22:30:39.843Z
Answers.UserDisplayName: David Hoelzer
Body: <p>I am developing my first feed-forward fully-connected ANN from scratch  based on batch learning mode on a toy training set. I am using back-propagation for calculating the gradient of the loss function with respect to weights and biases and using the gradient descent method as a learning rule.</p>
  <p>Unfortunately when I print the training loss value (sum of square), it gets bigger as the epoch increases as you can see here:</p>
  <pre><code>E(0) on TrS is: [[7.83898769]]
  E(1) on TrS is: [[10.00738465]]
  E(2) on TrS is: [[10.76653098]]
  E(3) on TrS is: [[15.94001008]]
  E(4) on TrS is: [[23.80650667]]
  E(5) on TrS is: [[28.65805023]]
  E(6) on TrS is: [[29.56550719]]
  E(7) on TrS is: [[30.5424694]]
  E(8) on TrS is: [[34.26980112]]
  E(9) on TrS is: [[39.9948856]]
  </code></pre>
  <p>This is my <code>loss_functions.py</code> file:</p>
  <pre><code>import numpy as np
  
  def sum_of_squares(c, t, y, derivative=False):
      ret = 0
      for k in range(c):
          ret += np.square(y - t)
      ret = 1 / 2 * ret
      if derivative:
          return y - t
      return ret
  </code></pre>
  <p>this is my <code>activation_functions.py</code> file:</p>
  <pre><code>import numpy as np
  
  
  def sigmoid(a, derivative=False):
      f_a = 1 / (1 + np.exp(-a))
      df_a = np.multiply(f_a, (1 - f_a)) 
      if derivative:
          return df_a
      return f_a
  
  
  def identity(a, derivative=False):
      f = a
      df = np.ones(np.shape(a))
      if derivative:
          return df
      return f
  </code></pre>
  <p>and this is the <code>main.py</code> file:</p>
  <pre><code>from activation_functions import *
  from loss_functions import *
  
  
  class NeuralNetwork:
  
      def __init__(self):
          self.layers = []
  
      def add_layer(self, layer):
          self.layers.append(layer)
  
      def create(self):
          for i, layer in enumerate(self.layers):
              if i == 0:
                  layer.type = &quot;input&quot;
              else:
                  if i == len(self.layers) - 1:
                      layer.type = &quot;output&quot;
                  else:
                      layer.type = &quot;hidden&quot;
                  layer.configure(self.layers[i - 1].neurons)
  
      def train(self, X, targets):
          MAX_EPOCHS = 10
          loss_function = sum_of_squares
          E = 0  # errore sull'intero DS
          for epoch in range(MAX_EPOCHS):
              for i, x in enumerate(X):
                  target = targets[i]
                  prediction = self.forward_prop(x.T)
                  E_n = loss_function(c, target, prediction)
                  E += E_n  
                  self.back_prop(target, local_loss=sum_of_squares)
              print(&quot;E(%d) on TrS is:&quot; % epoch, E)  # increasing!!!
              self.learning_rule(l_rate=0.05)  
  
      def forward_prop(self, z):
          for layer in self.layers:
              z = layer.forward_prop_step(z)
          return z
  
      def back_prop(self, target, local_loss):
          for i, layer in enumerate(self.layers[:0:-1]):
              next_layer = self.layers[-i]  
              prev_layer = self.layers[-i - 2] 
              layer.back_prop_step(next_layer, prev_layer, target, local_loss)
  
      def learning_rule(self, l_rate):
          # GD
          for layer in self.layers:
              if layer.type != &quot;input&quot;:
                  layer.weight -= l_rate * layer.dE_dW
                  layer.bias -= l_rate * layer.dE_db
  
  
  class Layer:
  
      def __init__(self, neurons, type=None, activation=None):
          self.dE_dW = 0 
          self.dE_db = 0
          self.dEn_db = None  # based on the n-th item
          self.dEn_dW = None  # based on the n-th item
          self.dact_a = None  
          self.out = None
          self.weight = None  
          self.bias = None 
          self.w_sum = None  
          self.neurons = neurons 
          self.type = type  
          self.activation = activation 
          self.deltas = None  
  
      def configure(self, prev_layer_neurons):
          self.weight = np.asmatrix(np.random.normal(0, 0.5, (self.neurons, prev_layer_neurons)))
          self.bias = np.asmatrix(np.random.normal(0, 0.5, self.neurons)).T 
          if self.activation is None:
              if self.type == &quot;hidden&quot;:
                  self.activation = sigmoid
              elif self.type == &quot;output&quot;:
                  self.activation = identity
  
      def forward_prop_step(self, z):
          if self.type == &quot;input&quot;:
              self.out = z
          else:
              self.w_sum = np.dot(self.weight, z) + self.bias
              self.out = self.activation(self.w_sum)
          return self.out
  
      def back_prop_step(self, next_layer, prev_layer, target, local_loss):
          if self.type == &quot;input&quot;:
              pass
          elif self.type == &quot;output&quot;:
              self.dact_a = self.activation(self.w_sum, derivative=True) 
              self.deltas = np.multiply(self.dact_a, local_loss(c, target, self.out, derivative=True)) 
          else:
              self.dact_a = self.activation(self.w_sum, derivative=True)  
              self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weight.T, next_layer.deltas))
  
          self.dEn_dW = np.dot(self.deltas, prev_layer.out.T)
  
          self.dEn_db = self.deltas
  
          self.dE_dW += self.dEn_dW
  
          self.dE_db += self.dEn_db
  
  
  if __name__ == '__main__':
      net = NeuralNetwork() 
  
      for m in (2, 4, 4, 1):
          net.add_layer(Layer(m))
  
      net.create()
  
      X = np.asmatrix([
          [1, 0],
          [1, 1],
          [0, 1],
          [0, 0]
      ])
  
      targets = np.asarray([1, 0, 0, 0])
  
      net.train(X, targets)  
  </code></pre>
  <p>What I did for trying so far to fix result:</p>
  <ol>
  <li>Check for any bug i code</li>
  <li>Decrease the learning rate (<code>l_rate</code>)</li>
  <li>Increase <code>MAX_EPOCHS</code> value</li>
  <li>Replace <code>-</code> symbol to <code>+</code> in GD formula</li>
  <li>Loss calculation with cross entropy method</li>
  </ol>
  <p>Unfortunately none of these worked. There must be issue(s) in dataset (size), hyper parameter, neural network structure or implementation...</p>
  <p>Any suggestion appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-22T22:28:17.097Z
Comments.Text: What is the reason behind choose sum of square instead of cross entropy for your loss function? Also have you tried with available software like Keras with same input and output with large epoch like 300?
Comments.UserDisplayName: Cloud Cho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-23T08:42:32.250Z
Comments.Text: @CloudCho the problem persists if I change the loss function, as well
Comments.UserDisplayName: tail
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-23T18:45:01.367Z
Comments.Text: Thanks for information. I also wonder if your input (and first layer of neural network) too small. How about around 50 inputs instead of 4?
Comments.UserDisplayName: Cloud Cho
Tags: <neural-networks><training><python>
Title: Training Loss Value Increasing instead of Decreasing
UserDisplayName: tail
ThreadText: # Training Loss Value Increasing instead of Decreasing
  
  (Asked by: username_0 on 2022-09-21)
  
  I am developing my first feed-forward fully-connected ANN from scratch  based on batch learning mode on a toy training set. I am using back-propagation for calculating the gradient of the loss function with respect to weights and biases and using the gradient descent method as a learning rule.
  Unfortunately when I print the training loss value (sum of square), it gets bigger as the epoch increases as you can see here:
  E(0) on TrS is: [[7.83898769]]
  E(1) on TrS is: [[10.00738465]]
  E(2) on TrS is: [[10.76653098]]
  E(3) on TrS is: [[15.94001008]]
  E(4) on TrS is: [[23.80650667]]
  E(5) on TrS is: [[28.65805023]]
  E(6) on TrS is: [[29.56550719]]
  E(7) on TrS is: [[30.5424694]]
  E(8) on TrS is: [[34.26980112]]
  E(9) on TrS is: [[39.9948856]]
  
  This is my loss_functions.py file:
  import numpy as np
  
  def sum_of_squares(c, t, y, derivative=False):
      ret = 0
      for k in range(c):
          ret += np.square(y - t)
      ret = 1 / 2 * ret
      if derivative:
          return y - t
      return ret
  
  this is my activation_functions.py file:
  import numpy as np
  
  
  def sigmoid(a, derivative=False):
      f_a = 1 / (1 + np.exp(-a))
      df_a = np.multiply(f_a, (1 - f_a)) 
      if derivative:
          return df_a
      return f_a
  
  
  def identity(a, derivative=False):
      f = a
      df = np.ones(np.shape(a))
      if derivative:
          return df
      return f
  
  and this is the main.py file:
  from activation_functions import *
  from loss_functions import *
  
  
  class NeuralNetwork:
  
      def __init__(self):
          self.layers = []
  
      def add_layer(self, layer):
          self.layers.append(layer)
  
      def create(self):
          for i, layer in enumerate(self.layers):
              if i == 0:
                  layer.type = "input"
              else:
                  if i == len(self.layers) - 1:
                      layer.type = "output"
                  else:
                      layer.type = "hidden"
                  layer.configure(self.layers[i - 1].neurons)
  
      def train(self, X, targets):
          MAX_EPOCHS = 10
          loss_function = sum_of_squares
          E = 0  # errore sull'intero DS
          for epoch in range(MAX_EPOCHS):
              for i, x in enumerate(X):
                  target = targets[i]
                  prediction = self.forward_prop(x.T)
                  E_n = loss_function(c, target, prediction)
                  E += E_n  
                  self.back_prop(target, local_loss=sum_of_squares)
              print("E(%d) on TrS is:" % epoch, E)  # increasing!!!
              self.learning_rule(l_rate=0.05)  
  
      def forward_prop(self, z):
          for layer in self.layers:
              z = layer.forward_prop_step(z)
          return z
  
      def back_prop(self, target, local_loss):
          for i, layer in enumerate(self.layers[:0:-1]):
              next_layer = self.layers[-i]  
              prev_layer = self.layers[-i - 2] 
              layer.back_prop_step(next_layer, prev_layer, target, local_loss)
  
      def learning_rule(self, l_rate):
          # GD
          for layer in self.layers:
              if layer.type != "input":
                  layer.weight -= l_rate * layer.dE_dW
                  layer.bias -= l_rate * layer.dE_db
  
  
  class Layer:
  
      def __init__(self, neurons, type=None, activation=None):
          self.dE_dW = 0 
          self.dE_db = 0
          self.dEn_db = None  # based on the n-th item
          self.dEn_dW = None  # based on the n-th item
          self.dact_a = None  
          self.out = None
          self.weight = None  
          self.bias = None 
          self.w_sum = None  
          self.neurons = neurons 
          self.type = type  
          self.activation = activation 
          self.deltas = None  
  
      def configure(self, prev_layer_neurons):
          self.weight = np.asmatrix(np.random.normal(0, 0.5, (self.neurons, prev_layer_neurons)))
          self.bias = np.asmatrix(np.random.normal(0, 0.5, self.neurons)).T 
          if self.activation is None:
              if self.type == "hidden":
                  self.activation = sigmoid
              elif self.type == "output":
                  self.activation = identity
  
      def forward_prop_step(self, z):
          if self.type == "input":
              self.out = z
          else:
              self.w_sum = np.dot(self.weight, z) + self.bias
              self.out = self.activation(self.w_sum)
          return self.out
  
      def back_prop_step(self, next_layer, prev_layer, target, local_loss):
          if self.type == "input":
              pass
          elif self.type == "output":
              self.dact_a = self.activation(self.w_sum, derivative=True) 
              self.deltas = np.multiply(self.dact_a, local_loss(c, target, self.out, derivative=True)) 
          else:
              self.dact_a = self.activation(self.w_sum, derivative=True)  
              self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weight.T, next_layer.deltas))
  
          self.dEn_dW = np.dot(self.deltas, prev_layer.out.T)
  
          self.dEn_db = self.deltas
  
          self.dE_dW += self.dEn_dW
  
          self.dE_db += self.dEn_db
  
  
  if __name__ == '__main__':
      net = NeuralNetwork() 
  
      for m in (2, 4, 4, 1):
          net.add_layer(Layer(m))
  
      net.create()
  
      X = np.asmatrix([
          [1, 0],
          [1, 1],
          [0, 1],
          [0, 0]
      ])
  
      targets = np.asarray([1, 0, 0, 0])
  
      net.train(X, targets)  
  
  What I did for trying so far to fix result:
  
  Check for any bug i code
  Decrease the learning rate (l_rate)
  Increase MAX_EPOCHS value
  Replace - symbol to + in GD formula
  Loss calculation with cross entropy method
  
  Unfortunately none of these worked. There must be issue(s) in dataset (size), hyper parameter, neural network structure or implementation...
  Any suggestion appreciated.
  
  **Tags:** neural-networks, training, python
  > **Comments:**
  > **username_1**: What is the reason behind choose sum of square instead of cross entropy for your loss function? Also have you tried with available software like Keras with same input and output with large epoch like 300?  
  > **username_0**: @CloudCho the problem persists if I change the loss function, as well  
  > **username_1**: Thanks for information. I also wonder if your input (and first layer of neural network) too small. How about around 50 inputs instead of 4?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-09-21. Score: 2
  
  It appears that you are adding the partial derivates (gradient) to make adjustments rather than subtracting them. Remember that the gradient vector points "upward." You may find things improve dramatically if you subtract them (head downward) to perform gradient *descent*.
  
  > **Comments:**
  > **username_0**: Then you think my formula for the gradient descent is incorrect and I need to replace - with +? In that case, the problem persists  
  > **username_2**: Your learning rate is also quite large in my opinion.  
  > **username_0**: The problem persists for `l_rate = 0.000005` as well  
  > **username_0**: I think there must be a bug somewhere in the code, but I cannot spot where yet  
  
  ---
  
  