<p>Your backward differentiation does not seem to follow the forward computation.</p>
<p>I prefer marking the gradient (row vector) with a letter <code>g</code> (in AD literature also <code>b</code> like TeX <code>\bar</code>) before the variable name, for tangent direction <code>d</code> (like direction or TeX <code>\dot</code>).</p>
<p>Beginning from the last step, one should get</p>
<ul>
<li>from the residual <code>0.5*sum((A-Y)**2)</code> indeed <code>gA2 = (A2-Y).T</code></li>
<li>from the last step <code>A2 = sigmoid(Z2)</code> you should get <code>gZ2 = gA2*Dsigmoid(Z2)</code> (component-wise product)</li>
<li>Next up is <code>Z2 = W2.dot(A1) + b2</code>. Using generic directions <code>dA1</code>, <code>dW2</code> (column vectors) etc., the defining relation is
<pre><code>gZ2 @ dZ2 = gZ2 @ dW2 @ A1 + gZ2 @ W2 @ dA1 + gZ2 @ db2
           = trace(gW2 @ dW2) + gA1 @ dA1 + gb2 * db2
</code></pre>
which implies
<pre><code>gA1 = gZ2 @ W2
gW2 = A1 @ gZ2 # this is a matrix as product column times row
gb2 = gZ2
</code></pre>
</li>
<li>next <code>A1 = relu(Z1)</code> leads to <code>gZ1 = gA1 * Drelu(Z1)</code> (component-wise)</li>
<li>finally <code>Z1 = W1.dot(X) + b1</code> similar to above
<pre><code>gX = gZ1 @ W1
gW1 = X @ gZ1
gb1 = gZ1
</code></pre>
</li>
</ul>
<p>In total, you need some slight modifications in the backward iteration</p>
<pre><code>def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):

    gA2 = (A2 - Y).T
    gZ2 = gA2 * sigmoid_prime(Z2.T)   # Z2*(1-Z2)

    gA1 = gZ2 @ W2
    gW2 = A1 @ gZ2
    gb2 = gZ2

    gZ1 = gA1 * relu_prime(Z1.T)       # 0.5*(1+signum(Z1))

    # gX = gZ1 @ W1
    gW1 = X @ gZ1
    gb1 = gZ1

    return gW1.T, gW2.T, gb1.T, gb2.T
</code></pre>
<p>Or with the gradients and every equation transposed to above</p>
<pre><code>def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):

    gA2 = A2 - Y
    gZ2 = sigmoid_prime(Z2) * gA2   # Z2*(1-Z2)

    gA1 = W2.T @ gZ2
    gW2 = gZ2 @ A1.T
    gb2 = gZ2

    gZ1 = relu_prime(Z1) * gA1       # 0.5*(1+signum(Z1))

    # gX = W1.T @ gZ1
    gW1 = gZ1 @ X.T
    gb1 = gZ1

    return gW1, gW2, gb1, gb2
</code></pre>

CC BY-SA 4.0
2022-11-22T13:40:55.070Z
I found out I was passing incorrect input data to my forwarrd propagation function. After changing it to `each_example` it did work start to work with increasing accuracy over time. Now I never got accuracy of over .95 with the architecture but I'll take it. However, reading  your comment, I gather I might have  made some errs in back_prop, I'll look into it. I felt pretty confident about it because all the equations came from Ng's Stanford ML lecture. I followed along derived them myself using partial derv.  Only Relu_prime came up during the back_prop in my case.
Maks
CC BY-SA 4.0
2022-11-22T11:41:01.467Z
2022-11-22T11:47:40.867Z
2022-11-22T11:47:40.867Z
Lutz Lehmann
<p>I am coding a classifier neural network from scratch. It is not really learning and I believe that somewhere there is a gradient explosion/vanishing issue. Could be some other stuff as well that I cannot imagine right now.</p>
<p>I have coded my own 2000 samples data set that has two features: x1, x2 and a label column that has 0 or 1.</p>
<p>I have tested the architecture on a neural network that I made via keras framework and it yielded an  85%  accuracy on the same dataset with same epoch value. Its fine that accuracy  was 0.85, thing is it worked.</p>
<p>Please help me figure out what am I doing wrong in my code below. Thank you!</p>
<p>My code:</p>
<pre><code>import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from my_first_nnfs_dataset import data_df

df = data_df

df = df.reset_index(drop = True)

len_of_training_data = 1900

max_df = df.max()['data_y']

X_train = np.array(df[:len_of_training_data][['data_x','data_y']]/max_df).T * 10
y_train = np.array(df[:len_of_training_data][['label']]).T

X_test = np.array(df[len_of_training_data:][['data_x','data_y']]/max_df).T  * 10
y_test = np.array(df[len_of_training_data:][['label']]).T



def initialize_parameters():

    W1 = np.random.rand(3,2)
    b1 = np.random.rand(3,1)

    W2 = np.random.rand(2,3)
    b2 = np.random.rand(2,1)

    return W1, b1, W2, b2

def relu(X):
    return np.maximum(0, X)

def relu_prime(X):
    return X &gt; 0

def sigmoid(X):
    return 1/(1 + np.exp(-X))

def forward_propagation(W1, b1, W2, b2, X):

    Z1 = W1.dot(X) + b1
    A1 = relu(Z1)
    Z2 = W2.dot(A1) + b2
    A2 = sigmoid(Z2)

    return Z1, A1, Z2, A2


def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):

    a = A2 - Y
    b = a.dot(A1.T)
    dW2 = b

    c = W2.T.dot(a)
    d = np.multiply(c, relu_prime(Z1))
    e = d.dot(X.T)
    dW1 = e

    db2 = np.sum(a)
    db1 = np.sum(d)


    return dW1, dW2, db1, db2

def update_parameters(W1, b1, W2, b2, dW1, dW2, db1, db2, alpha):


    W2 = W2 - alpha * dW2
    W1 = W1 - alpha * dW1
    b2 = b2 - alpha * db2
    b1 = b1 - alpha * db1

    return W1, b1, W2, b2


def one_hot_y(Y):
    one_hot_y = np.zeros((2, len_of_training_data))
    for i in range(0, y_train.size):
    
        if y_train[0,i] == 0:
            one_hot_y[0,i] = 1
        
        elif y_train[0,i] == 1:
            one_hot_y[1,i] = 1
    return one_hot_y

one_hot_y_train = one_hot_y(y_train)

a2_predictions = []


def accuracy(a2_predictions):
    a2_p = a2_predictions[-len_of_training_data:]
    latest_epoch = a2_p[-1]

    a = 0

    for i in range(y_train.size):
        if np.argmax(latest_epoch[:,i], axis = 0) == np.argmax(one_hot_y_train[:,i], axis = 0):
            a += 1
    return a/y_train.size

    
def train(X_train, one_hot_y_train, alpha, epoch):

    W1, b1, W2, b2 = initialize_parameters()
    for epoch in range(epochs):
    
        for column in range(y_train.size):
        
            each_example = X_train[:,column].reshape(2,1)
            each_one_hot_y = one_hot_y_train[:,column].reshape(2,1)
        
        
        
            Z1, A1, Z2, A2 = forward_propagation(W1, b1, W2, b2, X_train)
        
            dW1, dW2, db1, db2 = backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X_train, each_one_hot_y)
        
            W1, b1, W2, b2 = update_parameters(W1, b1, W2, b2, dW1, dW2, db1, db2, alpha)
        
            a2_predictions.append(A2)
        
       
        
        if epoch % 10 == 0:
        
            print(f'Epoch: {epoch}')
            print(f'Accuracy:{accuracy(a2_predictions)}\n')
        
    return W1, b1, W2, b2

epochs = 100
alpha = 0.1

W1, b1, W2, b2 = train(X_train, one_hot_y_train, alpha = alpha, epoch = epochs)

Z1, A1, Z2, A2 = forward_propagation(W1, b1, W2, b2, X_test)

test = np.zeros((1, y_test.size))

for i in range(y_test.size):
    if A2[0,i] &gt; A2[1,i]:
        test[0,i] = 0
    else:
        test[0,i] = 1
acc = 0

for i in range(len(test)):
    if test[0][i] == y_test[i][0]:
        acc += 1

print(f'accuracy: {acc/y_test.size}')
</code></pre>
<p>Output:</p>
<pre><code>/Users/apple/Desktop/my_first_nnfs.py:44: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-X))
Epoch: 0
Accuracy:0.5189473684210526

Epoch: 10
Accuracy:0.5189473684210526

Epoch: 20
Accuracy:0.5189473684210526

Epoch: 30
Accuracy:0.5189473684210526

Epoch: 40
Accuracy:0.5189473684210526

Epoch: 50
Accuracy:0.5189473684210526

Epoch: 60
Accuracy:0.5189473684210526

Epoch: 70
Accuracy:0.5189473684210526

Epoch: 80
Accuracy:0.5189473684210526

Epoch: 90
Accuracy:0.5189473684210526

accuracy: 0.009900990099009901
</code></pre>
<p>Necessary  variables after running:</p>
<pre><code>W1 = 0.914082   4.92167
     5.70267e+09    -1.40049e+10
    -0.986493   -8.28296

W2 = -61.9766   1.2412e+12  -85.8557
     8.91069    -1.2412e+12 16.2499


#A1 is all zeros array of shape (3,101)
#A2 is all ones array of shape (2,101)
</code></pre>
<p>PS - epoch = 1000  also has a very similar outcome.</p>

CC BY-SA 4.0
2022-11-22T03:19:18.603Z
this is just a friendly message to let you know that you may be better off posting your question on StackOverflow.  This question is more of a bug fix question, which is out of the scope for SE:AI.  Since you are new, you may not be aware of the what types of questions are suitable for this site.  This [article](https://ai.stackexchange.com/help/dont-ask) will help clarify.  This is probably why you have not received a response yet, and I just wanted to make sure that you get the help you need.  Best of luck!
Snehal Patel
<neural-networks><machine-learning><binary-classification><gradient>
I’m making a simple neural network from scratch and it won’t learn anything. Please help
Maks
# I’m making a simple neural network from scratch and it won’t learn anything. Please help

(Asked by: username_0 on 2022-11-21)

I am coding a classifier neural network from scratch. It is not really learning and I believe that somewhere there is a gradient explosion/vanishing issue. Could be some other stuff as well that I cannot imagine right now.
I have coded my own 2000 samples data set that has two features: x1, x2 and a label column that has 0 or 1.
I have tested the architecture on a neural network that I made via keras framework and it yielded an  85%  accuracy on the same dataset with same epoch value. Its fine that accuracy  was 0.85, thing is it worked.
Please help me figure out what am I doing wrong in my code below. Thank you!
My code:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from my_first_nnfs_dataset import data_df

df = data_df

df = df.reset_index(drop = True)

len_of_training_data = 1900

max_df = df.max()['data_y']

X_train = np.array(df[:len_of_training_data][['data_x','data_y']]/max_df).T * 10
y_train = np.array(df[:len_of_training_data][['label']]).T

X_test = np.array(df[len_of_training_data:][['data_x','data_y']]/max_df).T  * 10
y_test = np.array(df[len_of_training_data:][['label']]).T



def initialize_parameters():

    W1 = np.random.rand(3,2)
    b1 = np.random.rand(3,1)

    W2 = np.random.rand(2,3)
    b2 = np.random.rand(2,1)

    return W1, b1, W2, b2

def relu(X):
    return np.maximum(0, X)

def relu_prime(X):
    return X > 0

def sigmoid(X):
    return 1/(1 + np.exp(-X))

def forward_propagation(W1, b1, W2, b2, X):

    Z1 = W1.dot(X) + b1
    A1 = relu(Z1)
    Z2 = W2.dot(A1) + b2
    A2 = sigmoid(Z2)

    return Z1, A1, Z2, A2


def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):

    a = A2 - Y
    b = a.dot(A1.T)
    dW2 = b

    c = W2.T.dot(a)
    d = np.multiply(c, relu_prime(Z1))
    e = d.dot(X.T)
    dW1 = e

    db2 = np.sum(a)
    db1 = np.sum(d)


    return dW1, dW2, db1, db2

def update_parameters(W1, b1, W2, b2, dW1, dW2, db1, db2, alpha):


    W2 = W2 - alpha * dW2
    W1 = W1 - alpha * dW1
    b2 = b2 - alpha * db2
    b1 = b1 - alpha * db1

    return W1, b1, W2, b2


def one_hot_y(Y):
    one_hot_y = np.zeros((2, len_of_training_data))
    for i in range(0, y_train.size):
    
        if y_train[0,i] == 0:
            one_hot_y[0,i] = 1
        
        elif y_train[0,i] == 1:
            one_hot_y[1,i] = 1
    return one_hot_y

one_hot_y_train = one_hot_y(y_train)

a2_predictions = []


def accuracy(a2_predictions):
    a2_p = a2_predictions[-len_of_training_data:]
    latest_epoch = a2_p[-1]

    a = 0

    for i in range(y_train.size):
        if np.argmax(latest_epoch[:,i], axis = 0) == np.argmax(one_hot_y_train[:,i], axis = 0):
            a += 1
    return a/y_train.size

    
def train(X_train, one_hot_y_train, alpha, epoch):

    W1, b1, W2, b2 = initialize_parameters()
    for epoch in range(epochs):
    
        for column in range(y_train.size):
        
            each_example = X_train[:,column].reshape(2,1)
            each_one_hot_y = one_hot_y_train[:,column].reshape(2,1)
        
        
        
            Z1, A1, Z2, A2 = forward_propagation(W1, b1, W2, b2, X_train)
        
            dW1, dW2, db1, db2 = backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X_train, each_one_hot_y)
        
            W1, b1, W2, b2 = update_parameters(W1, b1, W2, b2, dW1, dW2, db1, db2, alpha)
        
            a2_predictions.append(A2)
        
       
        
        if epoch % 10 == 0:
        
            print(f'Epoch: {epoch}')
            print(f'Accuracy:{accuracy(a2_predictions)}\n')
        
    return W1, b1, W2, b2

epochs = 100
alpha = 0.1

W1, b1, W2, b2 = train(X_train, one_hot_y_train, alpha = alpha, epoch = epochs)

Z1, A1, Z2, A2 = forward_propagation(W1, b1, W2, b2, X_test)

test = np.zeros((1, y_test.size))

for i in range(y_test.size):
    if A2[0,i] > A2[1,i]:
        test[0,i] = 0
    else:
        test[0,i] = 1
acc = 0

for i in range(len(test)):
    if test[0][i] == y_test[i][0]:
        acc += 1

print(f'accuracy: {acc/y_test.size}')

Output:
/Users/apple/Desktop/my_first_nnfs.py:44: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-X))
Epoch: 0
Accuracy:0.5189473684210526

Epoch: 10
Accuracy:0.5189473684210526

Epoch: 20
Accuracy:0.5189473684210526

Epoch: 30
Accuracy:0.5189473684210526

Epoch: 40
Accuracy:0.5189473684210526

Epoch: 50
Accuracy:0.5189473684210526

Epoch: 60
Accuracy:0.5189473684210526

Epoch: 70
Accuracy:0.5189473684210526

Epoch: 80
Accuracy:0.5189473684210526

Epoch: 90
Accuracy:0.5189473684210526

accuracy: 0.009900990099009901

Necessary  variables after running:
W1 = 0.914082   4.92167
     5.70267e+09    -1.40049e+10
    -0.986493   -8.28296

W2 = -61.9766   1.2412e+12  -85.8557
     8.91069    -1.2412e+12 16.2499


#A1 is all zeros array of shape (3,101)
#A2 is all ones array of shape (2,101)

PS - epoch = 1000  also has a very similar outcome.

**Tags:** neural-networks, machine-learning, binary-classification, gradient

> **Comments:**
> **username_1**: this is just a friendly message to let you know that you may be better off posting your question on StackOverflow.  This question is more of a bug fix question, which is out of the scope for SE:AI.  Since you are new, you may not be aware of the what types of questions are suitable for this site.  This [article](https://ai.stackexchange.com/help/dont-ask) will help clarify.  This is probably why you have not received a response yet, and I just wanted to make sure that you get the help you need.  Best of luck!  

---

## 1 answers

### Answer by username_2 on 2022-11-22. Score: 1

Your backward differentiation does not seem to follow the forward computation.
I prefer marking the gradient (row vector) with a letter g (in AD literature also b like TeX \bar) before the variable name, for tangent direction d (like direction or TeX \dot).
Beginning from the last step, one should get

from the residual 0.5*sum((A-Y)**2) indeed gA2 = (A2-Y).T
from the last step A2 = sigmoid(Z2) you should get gZ2 = gA2*Dsigmoid(Z2) (component-wise product)
Next up is Z2 = W2.dot(A1) + b2. Using generic directions dA1, dW2 (column vectors) etc., the defining relation is
gZ2 @ dZ2 = gZ2 @ dW2 @ A1 + gZ2 @ W2 @ dA1 + gZ2 @ db2
           = trace(gW2 @ dW2) + gA1 @ dA1 + gb2 * db2

which implies
gA1 = gZ2 @ W2
gW2 = A1 @ gZ2 # this is a matrix as product column times row
gb2 = gZ2


next A1 = relu(Z1) leads to gZ1 = gA1 * Drelu(Z1) (component-wise)
finally Z1 = W1.dot(X) + b1 similar to above
gX = gZ1 @ W1
gW1 = X @ gZ1
gb1 = gZ1



In total, you need some slight modifications in the backward iteration
def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):

    gA2 = (A2 - Y).T
    gZ2 = gA2 * sigmoid_prime(Z2.T)   # Z2*(1-Z2)

    gA1 = gZ2 @ W2
    gW2 = A1 @ gZ2
    gb2 = gZ2

    gZ1 = gA1 * relu_prime(Z1.T)       # 0.5*(1+signum(Z1))

    # gX = gZ1 @ W1
    gW1 = X @ gZ1
    gb1 = gZ1

    return gW1.T, gW2.T, gb1.T, gb2.T

Or with the gradients and every equation transposed to above
def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):

    gA2 = A2 - Y
    gZ2 = sigmoid_prime(Z2) * gA2   # Z2*(1-Z2)

    gA1 = W2.T @ gZ2
    gW2 = gZ2 @ A1.T
    gb2 = gZ2

    gZ1 = relu_prime(Z1) * gA1       # 0.5*(1+signum(Z1))

    # gX = W1.T @ gZ1
    gW1 = gZ1 @ X.T
    gb1 = gZ1

    return gW1, gW2, gb1, gb2

> **Comments:**
> **username_0**: I found out I was passing incorrect input data to my forwarrd propagation function. After changing it to `each_example` it did work start to work with increasing accuracy over time. Now I never got accuracy of over .95 with the architecture but I'll take it. However, reading  your comment, I gather I might have  made some errs in back_prop, I'll look into it. I felt pretty confident about it because all the equations came from Ng's Stanford ML lecture. I followed along derived them myself using partial derv.  Only Relu_prime came up during the back_prop in my case.  

---

