<p>It appears that you are adding the partial derivates (gradient) to make adjustments rather than subtracting them.  Remember that the gradient vector points &quot;upward.&quot;  You may find things improve dramatically if you subtract them (head downward) to perform gradient <em>descent</em>.</p>

CC BY-SA 4.0
2022-09-22T07:29:14.557Z
Then you think my formula for the gradient descent is incorrect and I need to replace - with +? In that case, the problem persists
tail
CC BY-SA 4.0
2022-09-22T14:06:41.463Z
Your learning rate is also quite large in my opinion.
David Hoelzer
CC BY-SA 4.0
2022-09-22T14:16:10.870Z
The problem persists for `l_rate = 0.000005` as well
tail
CC BY-SA 4.0
2022-09-22T14:34:01.650Z
I think there must be a bug somewhere in the code, but I cannot spot where yet
tail
CC BY-SA 4.0
2022-09-21T22:30:39.843Z
2022-09-21T22:30:39.843Z
David Hoelzer
<p>I am developing my first feed-forward fully-connected ANN from scratch  based on batch learning mode on a toy training set. I am using back-propagation for calculating the gradient of the loss function with respect to weights and biases and using the gradient descent method as a learning rule.</p>
<p>Unfortunately when I print the training loss value (sum of square), it gets bigger as the epoch increases as you can see here:</p>
<pre><code>E(0) on TrS is: [[7.83898769]]
E(1) on TrS is: [[10.00738465]]
E(2) on TrS is: [[10.76653098]]
E(3) on TrS is: [[15.94001008]]
E(4) on TrS is: [[23.80650667]]
E(5) on TrS is: [[28.65805023]]
E(6) on TrS is: [[29.56550719]]
E(7) on TrS is: [[30.5424694]]
E(8) on TrS is: [[34.26980112]]
E(9) on TrS is: [[39.9948856]]
</code></pre>
<p>This is my <code>loss_functions.py</code> file:</p>
<pre><code>import numpy as np

def sum_of_squares(c, t, y, derivative=False):
    ret = 0
    for k in range(c):
        ret += np.square(y - t)
    ret = 1 / 2 * ret
    if derivative:
        return y - t
    return ret
</code></pre>
<p>this is my <code>activation_functions.py</code> file:</p>
<pre><code>import numpy as np


def sigmoid(a, derivative=False):
    f_a = 1 / (1 + np.exp(-a))
    df_a = np.multiply(f_a, (1 - f_a)) 
    if derivative:
        return df_a
    return f_a


def identity(a, derivative=False):
    f = a
    df = np.ones(np.shape(a))
    if derivative:
        return df
    return f
</code></pre>
<p>and this is the <code>main.py</code> file:</p>
<pre><code>from activation_functions import *
from loss_functions import *


class NeuralNetwork:

    def __init__(self):
        self.layers = []

    def add_layer(self, layer):
        self.layers.append(layer)

    def create(self):
        for i, layer in enumerate(self.layers):
            if i == 0:
                layer.type = &quot;input&quot;
            else:
                if i == len(self.layers) - 1:
                    layer.type = &quot;output&quot;
                else:
                    layer.type = &quot;hidden&quot;
                layer.configure(self.layers[i - 1].neurons)

    def train(self, X, targets):
        MAX_EPOCHS = 10
        loss_function = sum_of_squares
        E = 0  # errore sull'intero DS
        for epoch in range(MAX_EPOCHS):
            for i, x in enumerate(X):
                target = targets[i]
                prediction = self.forward_prop(x.T)
                E_n = loss_function(c, target, prediction)
                E += E_n  
                self.back_prop(target, local_loss=sum_of_squares)
            print(&quot;E(%d) on TrS is:&quot; % epoch, E)  # increasing!!!
            self.learning_rule(l_rate=0.05)  

    def forward_prop(self, z):
        for layer in self.layers:
            z = layer.forward_prop_step(z)
        return z

    def back_prop(self, target, local_loss):
        for i, layer in enumerate(self.layers[:0:-1]):
            next_layer = self.layers[-i]  
            prev_layer = self.layers[-i - 2] 
            layer.back_prop_step(next_layer, prev_layer, target, local_loss)

    def learning_rule(self, l_rate):
        # GD
        for layer in self.layers:
            if layer.type != &quot;input&quot;:
                layer.weight -= l_rate * layer.dE_dW
                layer.bias -= l_rate * layer.dE_db


class Layer:

    def __init__(self, neurons, type=None, activation=None):
        self.dE_dW = 0 
        self.dE_db = 0
        self.dEn_db = None  # based on the n-th item
        self.dEn_dW = None  # based on the n-th item
        self.dact_a = None  
        self.out = None
        self.weight = None  
        self.bias = None 
        self.w_sum = None  
        self.neurons = neurons 
        self.type = type  
        self.activation = activation 
        self.deltas = None  

    def configure(self, prev_layer_neurons):
        self.weight = np.asmatrix(np.random.normal(0, 0.5, (self.neurons, prev_layer_neurons)))
        self.bias = np.asmatrix(np.random.normal(0, 0.5, self.neurons)).T 
        if self.activation is None:
            if self.type == &quot;hidden&quot;:
                self.activation = sigmoid
            elif self.type == &quot;output&quot;:
                self.activation = identity

    def forward_prop_step(self, z):
        if self.type == &quot;input&quot;:
            self.out = z
        else:
            self.w_sum = np.dot(self.weight, z) + self.bias
            self.out = self.activation(self.w_sum)
        return self.out

    def back_prop_step(self, next_layer, prev_layer, target, local_loss):
        if self.type == &quot;input&quot;:
            pass
        elif self.type == &quot;output&quot;:
            self.dact_a = self.activation(self.w_sum, derivative=True) 
            self.deltas = np.multiply(self.dact_a, local_loss(c, target, self.out, derivative=True)) 
        else:
            self.dact_a = self.activation(self.w_sum, derivative=True)  
            self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weight.T, next_layer.deltas))

        self.dEn_dW = np.dot(self.deltas, prev_layer.out.T)

        self.dEn_db = self.deltas

        self.dE_dW += self.dEn_dW

        self.dE_db += self.dEn_db


if __name__ == '__main__':
    net = NeuralNetwork() 

    for m in (2, 4, 4, 1):
        net.add_layer(Layer(m))

    net.create()

    X = np.asmatrix([
        [1, 0],
        [1, 1],
        [0, 1],
        [0, 0]
    ])

    targets = np.asarray([1, 0, 0, 0])

    net.train(X, targets)  
</code></pre>
<p>What I did for trying so far to fix result:</p>
<ol>
<li>Check for any bug i code</li>
<li>Decrease the learning rate (<code>l_rate</code>)</li>
<li>Increase <code>MAX_EPOCHS</code> value</li>
<li>Replace <code>-</code> symbol to <code>+</code> in GD formula</li>
<li>Loss calculation with cross entropy method</li>
</ol>
<p>Unfortunately none of these worked. There must be issue(s) in dataset (size), hyper parameter, neural network structure or implementation...</p>
<p>Any suggestion appreciated.</p>

CC BY-SA 4.0
2022-09-22T22:28:17.097Z
What is the reason behind choose sum of square instead of cross entropy for your loss function? Also have you tried with available software like Keras with same input and output with large epoch like 300?
Cloud Cho
CC BY-SA 4.0
2022-09-23T08:42:32.250Z
@CloudCho the problem persists if I change the loss function, as well
tail
CC BY-SA 4.0
2022-09-23T18:45:01.367Z
Thanks for information. I also wonder if your input (and first layer of neural network) too small. How about around 50 inputs instead of 4?
Cloud Cho
<neural-networks><training><python>
Training Loss Value Increasing instead of Decreasing
tail
# Training Loss Value Increasing instead of Decreasing

(Asked by: username_0 on 2022-09-21)

I am developing my first feed-forward fully-connected ANN from scratch  based on batch learning mode on a toy training set. I am using back-propagation for calculating the gradient of the loss function with respect to weights and biases and using the gradient descent method as a learning rule.
Unfortunately when I print the training loss value (sum of square), it gets bigger as the epoch increases as you can see here:
E(0) on TrS is: [[7.83898769]]
E(1) on TrS is: [[10.00738465]]
E(2) on TrS is: [[10.76653098]]
E(3) on TrS is: [[15.94001008]]
E(4) on TrS is: [[23.80650667]]
E(5) on TrS is: [[28.65805023]]
E(6) on TrS is: [[29.56550719]]
E(7) on TrS is: [[30.5424694]]
E(8) on TrS is: [[34.26980112]]
E(9) on TrS is: [[39.9948856]]

This is my loss_functions.py file:
import numpy as np

def sum_of_squares(c, t, y, derivative=False):
    ret = 0
    for k in range(c):
        ret += np.square(y - t)
    ret = 1 / 2 * ret
    if derivative:
        return y - t
    return ret

this is my activation_functions.py file:
import numpy as np


def sigmoid(a, derivative=False):
    f_a = 1 / (1 + np.exp(-a))
    df_a = np.multiply(f_a, (1 - f_a)) 
    if derivative:
        return df_a
    return f_a


def identity(a, derivative=False):
    f = a
    df = np.ones(np.shape(a))
    if derivative:
        return df
    return f

and this is the main.py file:
from activation_functions import *
from loss_functions import *


class NeuralNetwork:

    def __init__(self):
        self.layers = []

    def add_layer(self, layer):
        self.layers.append(layer)

    def create(self):
        for i, layer in enumerate(self.layers):
            if i == 0:
                layer.type = "input"
            else:
                if i == len(self.layers) - 1:
                    layer.type = "output"
                else:
                    layer.type = "hidden"
                layer.configure(self.layers[i - 1].neurons)

    def train(self, X, targets):
        MAX_EPOCHS = 10
        loss_function = sum_of_squares
        E = 0  # errore sull'intero DS
        for epoch in range(MAX_EPOCHS):
            for i, x in enumerate(X):
                target = targets[i]
                prediction = self.forward_prop(x.T)
                E_n = loss_function(c, target, prediction)
                E += E_n  
                self.back_prop(target, local_loss=sum_of_squares)
            print("E(%d) on TrS is:" % epoch, E)  # increasing!!!
            self.learning_rule(l_rate=0.05)  

    def forward_prop(self, z):
        for layer in self.layers:
            z = layer.forward_prop_step(z)
        return z

    def back_prop(self, target, local_loss):
        for i, layer in enumerate(self.layers[:0:-1]):
            next_layer = self.layers[-i]  
            prev_layer = self.layers[-i - 2] 
            layer.back_prop_step(next_layer, prev_layer, target, local_loss)

    def learning_rule(self, l_rate):
        # GD
        for layer in self.layers:
            if layer.type != "input":
                layer.weight -= l_rate * layer.dE_dW
                layer.bias -= l_rate * layer.dE_db


class Layer:

    def __init__(self, neurons, type=None, activation=None):
        self.dE_dW = 0 
        self.dE_db = 0
        self.dEn_db = None  # based on the n-th item
        self.dEn_dW = None  # based on the n-th item
        self.dact_a = None  
        self.out = None
        self.weight = None  
        self.bias = None 
        self.w_sum = None  
        self.neurons = neurons 
        self.type = type  
        self.activation = activation 
        self.deltas = None  

    def configure(self, prev_layer_neurons):
        self.weight = np.asmatrix(np.random.normal(0, 0.5, (self.neurons, prev_layer_neurons)))
        self.bias = np.asmatrix(np.random.normal(0, 0.5, self.neurons)).T 
        if self.activation is None:
            if self.type == "hidden":
                self.activation = sigmoid
            elif self.type == "output":
                self.activation = identity

    def forward_prop_step(self, z):
        if self.type == "input":
            self.out = z
        else:
            self.w_sum = np.dot(self.weight, z) + self.bias
            self.out = self.activation(self.w_sum)
        return self.out

    def back_prop_step(self, next_layer, prev_layer, target, local_loss):
        if self.type == "input":
            pass
        elif self.type == "output":
            self.dact_a = self.activation(self.w_sum, derivative=True) 
            self.deltas = np.multiply(self.dact_a, local_loss(c, target, self.out, derivative=True)) 
        else:
            self.dact_a = self.activation(self.w_sum, derivative=True)  
            self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weight.T, next_layer.deltas))

        self.dEn_dW = np.dot(self.deltas, prev_layer.out.T)

        self.dEn_db = self.deltas

        self.dE_dW += self.dEn_dW

        self.dE_db += self.dEn_db


if __name__ == '__main__':
    net = NeuralNetwork() 

    for m in (2, 4, 4, 1):
        net.add_layer(Layer(m))

    net.create()

    X = np.asmatrix([
        [1, 0],
        [1, 1],
        [0, 1],
        [0, 0]
    ])

    targets = np.asarray([1, 0, 0, 0])

    net.train(X, targets)  

What I did for trying so far to fix result:

Check for any bug i code
Decrease the learning rate (l_rate)
Increase MAX_EPOCHS value
Replace - symbol to + in GD formula
Loss calculation with cross entropy method

Unfortunately none of these worked. There must be issue(s) in dataset (size), hyper parameter, neural network structure or implementation...
Any suggestion appreciated.

**Tags:** neural-networks, training, python

> **Comments:**
> **username_1**: What is the reason behind choose sum of square instead of cross entropy for your loss function? Also have you tried with available software like Keras with same input and output with large epoch like 300?  
> **username_0**: @CloudCho the problem persists if I change the loss function, as well  
> **username_1**: Thanks for information. I also wonder if your input (and first layer of neural network) too small. How about around 50 inputs instead of 4?  

---

## 1 answers

### Answer by username_2 on 2022-09-21. Score: 2

It appears that you are adding the partial derivates (gradient) to make adjustments rather than subtracting them. Remember that the gradient vector points "upward." You may find things improve dramatically if you subtract them (head downward) to perform gradient *descent*.

> **Comments:**
> **username_0**: Then you think my formula for the gradient descent is incorrect and I need to replace - with +? In that case, the problem persists  
> **username_2**: Your learning rate is also quite large in my opinion.  
> **username_0**: The problem persists for `l_rate = 0.000005` as well  
> **username_0**: I think there must be a bug somewhere in the code, but I cannot spot where yet  

---

