Answers.Body: <p>Your backward differentiation does not seem to follow the forward computation.</p>
  <p>I prefer marking the gradient (row vector) with a letter <code>g</code> (in AD literature also <code>b</code> like TeX <code>\bar</code>) before the variable name, for tangent direction <code>d</code> (like direction or TeX <code>\dot</code>).</p>
  <p>Beginning from the last step, one should get</p>
  <ul>
  <li>from the residual <code>0.5*sum((A-Y)**2)</code> indeed <code>gA2 = (A2-Y).T</code></li>
  <li>from the last step <code>A2 = sigmoid(Z2)</code> you should get <code>gZ2 = gA2*Dsigmoid(Z2)</code> (component-wise product)</li>
  <li>Next up is <code>Z2 = W2.dot(A1) + b2</code>. Using generic directions <code>dA1</code>, <code>dW2</code> (column vectors) etc., the defining relation is
  <pre><code>gZ2 @ dZ2 = gZ2 @ dW2 @ A1 + gZ2 @ W2 @ dA1 + gZ2 @ db2
             = trace(gW2 @ dW2) + gA1 @ dA1 + gb2 * db2
  </code></pre>
  which implies
  <pre><code>gA1 = gZ2 @ W2
  gW2 = A1 @ gZ2 # this is a matrix as product column times row
  gb2 = gZ2
  </code></pre>
  </li>
  <li>next <code>A1 = relu(Z1)</code> leads to <code>gZ1 = gA1 * Drelu(Z1)</code> (component-wise)</li>
  <li>finally <code>Z1 = W1.dot(X) + b1</code> similar to above
  <pre><code>gX = gZ1 @ W1
  gW1 = X @ gZ1
  gb1 = gZ1
  </code></pre>
  </li>
  </ul>
  <p>In total, you need some slight modifications in the backward iteration</p>
  <pre><code>def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):
  
      gA2 = (A2 - Y).T
      gZ2 = gA2 * sigmoid_prime(Z2.T)   # Z2*(1-Z2)
  
      gA1 = gZ2 @ W2
      gW2 = A1 @ gZ2
      gb2 = gZ2
  
      gZ1 = gA1 * relu_prime(Z1.T)       # 0.5*(1+signum(Z1))
  
      # gX = gZ1 @ W1
      gW1 = X @ gZ1
      gb1 = gZ1
  
      return gW1.T, gW2.T, gb1.T, gb2.T
  </code></pre>
  <p>Or with the gradients and every equation transposed to above</p>
  <pre><code>def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):
  
      gA2 = A2 - Y
      gZ2 = sigmoid_prime(Z2) * gA2   # Z2*(1-Z2)
  
      gA1 = W2.T @ gZ2
      gW2 = gZ2 @ A1.T
      gb2 = gZ2
  
      gZ1 = relu_prime(Z1) * gA1       # 0.5*(1+signum(Z1))
  
      # gX = W1.T @ gZ1
      gW1 = gZ1 @ X.T
      gb1 = gZ1
  
      return gW1, gW2, gb1, gb2
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-22T13:40:55.070Z
Answers.Text: I found out I was passing incorrect input data to my forwarrd propagation function. After changing it to `each_example` it did work start to work with increasing accuracy over time. Now I never got accuracy of over .95 with the architecture but I'll take it. However, reading  your comment, I gather I might have  made some errs in back_prop, I'll look into it. I felt pretty confident about it because all the equations came from Ng's Stanford ML lecture. I followed along derived them myself using partial derv.  Only Relu_prime came up during the back_prop in my case.
Answers.UserDisplayName: Maks
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-22T11:41:01.467Z
Answers.LastActivityDate: 2022-11-22T11:47:40.867Z
Answers.LastEditDate: 2022-11-22T11:47:40.867Z
Answers.UserDisplayName: Lutz Lehmann
Body: <p>I am coding a classifier neural network from scratch. It is not really learning and I believe that somewhere there is a gradient explosion/vanishing issue. Could be some other stuff as well that I cannot imagine right now.</p>
  <p>I have coded my own 2000 samples data set that has two features: x1, x2 and a label column that has 0 or 1.</p>
  <p>I have tested the architecture on a neural network that I made via keras framework and it yielded an  85%  accuracy on the same dataset with same epoch value. Its fine that accuracy  was 0.85, thing is it worked.</p>
  <p>Please help me figure out what am I doing wrong in my code below. Thank you!</p>
  <p>My code:</p>
  <pre><code>import numpy as np
  import pandas as pd
  import seaborn as sns
  import matplotlib.pyplot as plt
  from my_first_nnfs_dataset import data_df
  
  df = data_df
  
  df = df.reset_index(drop = True)
  
  len_of_training_data = 1900
  
  max_df = df.max()['data_y']
  
  X_train = np.array(df[:len_of_training_data][['data_x','data_y']]/max_df).T * 10
  y_train = np.array(df[:len_of_training_data][['label']]).T
  
  X_test = np.array(df[len_of_training_data:][['data_x','data_y']]/max_df).T  * 10
  y_test = np.array(df[len_of_training_data:][['label']]).T
  
  
  
  def initialize_parameters():
  
      W1 = np.random.rand(3,2)
      b1 = np.random.rand(3,1)
  
      W2 = np.random.rand(2,3)
      b2 = np.random.rand(2,1)
  
      return W1, b1, W2, b2
  
  def relu(X):
      return np.maximum(0, X)
  
  def relu_prime(X):
      return X &gt; 0
  
  def sigmoid(X):
      return 1/(1 + np.exp(-X))
  
  def forward_propagation(W1, b1, W2, b2, X):
  
      Z1 = W1.dot(X) + b1
      A1 = relu(Z1)
      Z2 = W2.dot(A1) + b2
      A2 = sigmoid(Z2)
  
      return Z1, A1, Z2, A2
  
  
  def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):
  
      a = A2 - Y
      b = a.dot(A1.T)
      dW2 = b
  
      c = W2.T.dot(a)
      d = np.multiply(c, relu_prime(Z1))
      e = d.dot(X.T)
      dW1 = e
  
      db2 = np.sum(a)
      db1 = np.sum(d)
  
  
      return dW1, dW2, db1, db2
  
  def update_parameters(W1, b1, W2, b2, dW1, dW2, db1, db2, alpha):
  
  
      W2 = W2 - alpha * dW2
      W1 = W1 - alpha * dW1
      b2 = b2 - alpha * db2
      b1 = b1 - alpha * db1
  
      return W1, b1, W2, b2
  
  
  def one_hot_y(Y):
      one_hot_y = np.zeros((2, len_of_training_data))
      for i in range(0, y_train.size):
      
          if y_train[0,i] == 0:
              one_hot_y[0,i] = 1
          
          elif y_train[0,i] == 1:
              one_hot_y[1,i] = 1
      return one_hot_y
  
  one_hot_y_train = one_hot_y(y_train)
  
  a2_predictions = []
  
  
  def accuracy(a2_predictions):
      a2_p = a2_predictions[-len_of_training_data:]
      latest_epoch = a2_p[-1]
  
      a = 0
  
      for i in range(y_train.size):
          if np.argmax(latest_epoch[:,i], axis = 0) == np.argmax(one_hot_y_train[:,i], axis = 0):
              a += 1
      return a/y_train.size
  
      
  def train(X_train, one_hot_y_train, alpha, epoch):
  
      W1, b1, W2, b2 = initialize_parameters()
      for epoch in range(epochs):
      
          for column in range(y_train.size):
          
              each_example = X_train[:,column].reshape(2,1)
              each_one_hot_y = one_hot_y_train[:,column].reshape(2,1)
          
          
          
              Z1, A1, Z2, A2 = forward_propagation(W1, b1, W2, b2, X_train)
          
              dW1, dW2, db1, db2 = backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X_train, each_one_hot_y)
          
              W1, b1, W2, b2 = update_parameters(W1, b1, W2, b2, dW1, dW2, db1, db2, alpha)
          
              a2_predictions.append(A2)
          
         
          
          if epoch % 10 == 0:
          
              print(f'Epoch: {epoch}')
              print(f'Accuracy:{accuracy(a2_predictions)}\n')
          
      return W1, b1, W2, b2
  
  epochs = 100
  alpha = 0.1
  
  W1, b1, W2, b2 = train(X_train, one_hot_y_train, alpha = alpha, epoch = epochs)
  
  Z1, A1, Z2, A2 = forward_propagation(W1, b1, W2, b2, X_test)
  
  test = np.zeros((1, y_test.size))
  
  for i in range(y_test.size):
      if A2[0,i] &gt; A2[1,i]:
          test[0,i] = 0
      else:
          test[0,i] = 1
  acc = 0
  
  for i in range(len(test)):
      if test[0][i] == y_test[i][0]:
          acc += 1
  
  print(f'accuracy: {acc/y_test.size}')
  </code></pre>
  <p>Output:</p>
  <pre><code>/Users/apple/Desktop/my_first_nnfs.py:44: RuntimeWarning: overflow encountered in exp
    return 1/(1 + np.exp(-X))
  Epoch: 0
  Accuracy:0.5189473684210526
  
  Epoch: 10
  Accuracy:0.5189473684210526
  
  Epoch: 20
  Accuracy:0.5189473684210526
  
  Epoch: 30
  Accuracy:0.5189473684210526
  
  Epoch: 40
  Accuracy:0.5189473684210526
  
  Epoch: 50
  Accuracy:0.5189473684210526
  
  Epoch: 60
  Accuracy:0.5189473684210526
  
  Epoch: 70
  Accuracy:0.5189473684210526
  
  Epoch: 80
  Accuracy:0.5189473684210526
  
  Epoch: 90
  Accuracy:0.5189473684210526
  
  accuracy: 0.009900990099009901
  </code></pre>
  <p>Necessary  variables after running:</p>
  <pre><code>W1 = 0.914082   4.92167
       5.70267e+09    -1.40049e+10
      -0.986493   -8.28296
  
  W2 = -61.9766   1.2412e+12  -85.8557
       8.91069    -1.2412e+12 16.2499
  
  
  #A1 is all zeros array of shape (3,101)
  #A2 is all ones array of shape (2,101)
  </code></pre>
  <p>PS - epoch = 1000  also has a very similar outcome.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-22T03:19:18.603Z
Comments.Text: this is just a friendly message to let you know that you may be better off posting your question on StackOverflow.  This question is more of a bug fix question, which is out of the scope for SE:AI.  Since you are new, you may not be aware of the what types of questions are suitable for this site.  This [article](https://ai.stackexchange.com/help/dont-ask) will help clarify.  This is probably why you have not received a response yet, and I just wanted to make sure that you get the help you need.  Best of luck!
Comments.UserDisplayName: Snehal Patel
Tags: <neural-networks><machine-learning><binary-classification><gradient>
Title: I’m making a simple neural network from scratch and it won’t learn anything. Please help
UserDisplayName: Maks
ThreadText: # I’m making a simple neural network from scratch and it won’t learn anything. Please help
  
  (Asked by: username_0 on 2022-11-21)
  
  I am coding a classifier neural network from scratch. It is not really learning and I believe that somewhere there is a gradient explosion/vanishing issue. Could be some other stuff as well that I cannot imagine right now.
  
  I have coded my own 2000 samples data set that has two features: x1, x2 and a label column that has 0 or 1.
  
  I have tested the architecture on a neural network that I made via keras framework and it yielded an 85% accuracy on the same dataset with same epoch value. Its fine that accuracy was 0.85, thing is it worked.
  
  Please help me figure out what am I doing wrong in my code below. Thank you!
  
  My code:
  
  ```
  import numpy as np
  import pandas as pd
  import seaborn as sns
  import matplotlib.pyplot as plt
  from my_first_nnfs_dataset import data_df
  
  df = data_df
  
  df = df.reset_index(drop = True)
  
  len_of_training_data = 1900
  
  max_df = df.max()['data_y']
  
  X_train = np.array(df[:len_of_training_data][['data_x','data_y']]/max_df).T * 10
  y_train = np.array(df[:len_of_training_data][['label']]).T
  
  X_test = np.array(df[len_of_training_data:][['data_x','data_y']]/max_df).T  * 10
  y_test = np.array(df[len_of_training_data:][['label']]).T
  
  def initialize_parameters():
  
      W1 = np.random.rand(3,2)
      b1 = np.random.rand(3,1)
  
      W2 = np.random.rand(2,3)
      b2 = np.random.rand(2,1)
  
      return W1, b1, W2, b2
  
  def relu(X):
      return np.maximum(0, X)
  
  def relu_prime(X):
      return X > 0
  
  def sigmoid(X):
      return 1/(1 + np.exp(-X))
  
  def forward_propagation(W1, b1, W2, b2, X):
  
      Z1 = W1.dot(X) + b1
      A1 = relu(Z1)
      Z2 = W2.dot(A1) + b2
      A2 = sigmoid(Z2)
  
      return Z1, A1, Z2, A2
  
  def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):
  
      a = A2 - Y
      b = a.dot(A1.T)
      dW2 = b
  
      c = W2.T.dot(a)
      d = np.multiply(c, relu_prime(Z1))
      e = d.dot(X.T)
      dW1 = e
  
      db2 = np.sum(a)
      db1 = np.sum(d)
  
      return dW1, dW2, db1, db2
  
  def update_parameters(W1, b1, W2, b2, dW1, dW2, db1, db2, alpha):
  
      W2 = W2 - alpha * dW2
      W1 = W1 - alpha * dW1
      b2 = b2 - alpha * db2
      b1 = b1 - alpha * db1
  
      return W1, b1, W2, b2
  
  def one_hot_y(Y):
      one_hot_y = np.zeros((2, len_of_training_data))
      for i in range(0, y_train.size):
      
          if y_train[0,i] == 0:
              one_hot_y[0,i] = 1
          
          elif y_train[0,i] == 1:
              one_hot_y[1,i] = 1
      return one_hot_y
  
  one_hot_y_train = one_hot_y(y_train)
  
  a2_predictions = []
  
  def accuracy(a2_predictions):
      a2_p = a2_predictions[-len_of_training_data:]
      latest_epoch = a2_p[-1]
  
      a = 0
  
      for i in range(y_train.size):
          if np.argmax(latest_epoch[:,i], axis = 0) == np.argmax(one_hot_y_train[:,i], axis = 0):
              a += 1
      return a/y_train.size
  
      
  def train(X_train, one_hot_y_train, alpha, epoch):
  
      W1, b1, W2, b2 = initialize_parameters()
      for epoch in range(epochs):
      
          for column in range(y_train.size):
          
              each_example = X_train[:,column].reshape(2,1)
              each_one_hot_y = one_hot_y_train[:,column].reshape(2,1)
          
          
          
              Z1, A1, Z2, A2 = forward_propagation(W1, b1, W2, b2, X_train)
          
              dW1, dW2, db1, db2 = backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X_train, each_one_hot_y)
          
              W1, b1, W2, b2 = update_parameters(W1, b1, W2, b2, dW1, dW2, db1, db2, alpha)
          
              a2_predictions.append(A2)
          
         
          
          if epoch % 10 == 0:
          
              print(f'Epoch: {epoch}')
              print(f'Accuracy:{accuracy(a2_predictions)}\n')
          
      return W1, b1, W2, b2
  
  epochs = 100
  alpha = 0.1
  
  W1, b1, W2, b2 = train(X_train, one_hot_y_train, alpha = alpha, epoch = epochs)
  
  Z1, A1, Z2, A2 = forward_propagation(W1, b1, W2, b2, X_test)
  
  test = np.zeros((1, y_test.size))
  
  for i in range(y_test.size):
      if A2[0,i] > A2[1,i]:
          test[0,i] = 0
      else:
          test[0,i] = 1
  acc = 0
  
  for i in range(len(test)):
      if test[0][i] == y_test[i][0]:
          acc += 1
  
  print(f'accuracy: {acc/y_test.size}')
  
  ```
  
  Output:
  
  ```
  /Users/apple/Desktop/my_first_nnfs.py:44: RuntimeWarning: overflow encountered in exp
    return 1/(1 + np.exp(-X))
  Epoch: 0
  Accuracy:0.5189473684210526
  
  Epoch: 10
  Accuracy:0.5189473684210526
  
  Epoch: 20
  Accuracy:0.5189473684210526
  
  Epoch: 30
  Accuracy:0.5189473684210526
  
  Epoch: 40
  Accuracy:0.5189473684210526
  
  Epoch: 50
  Accuracy:0.5189473684210526
  
  Epoch: 60
  Accuracy:0.5189473684210526
  
  Epoch: 70
  Accuracy:0.5189473684210526
  
  Epoch: 80
  Accuracy:0.5189473684210526
  
  Epoch: 90
  Accuracy:0.5189473684210526
  
  accuracy: 0.009900990099009901
  
  ```
  
  Necessary variables after running:
  
  ```
  W1 = 0.914082   4.92167
       5.70267e+09    -1.40049e+10
      -0.986493   -8.28296
  
  W2 = -61.9766   1.2412e+12  -85.8557
       8.91069    -1.2412e+12 16.2499
  
  #A1 is all zeros array of shape (3,101)
  #A2 is all ones array of shape (2,101)
  
  ```
  
  PS - epoch = 1000 also has a very similar outcome.
  
  **Tags:** neural-networks, machine-learning, binary-classification, gradient
  
  > **Comments:**
  > **username_1**: this is just a friendly message to let you know that you may be better off posting your question on StackOverflow.  This question is more of a bug fix question, which is out of the scope for SE:AI.  Since you are new, you may not be aware of the what types of questions are suitable for this site.  This [article](https://ai.stackexchange.com/help/dont-ask) will help clarify.  This is probably why you have not received a response yet, and I just wanted to make sure that you get the help you need.  Best of luck!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-11-22. Score: 1
  
  Your backward differentiation does not seem to follow the forward computation.
  
  I prefer marking the gradient (row vector) with a letter `g` (in AD literature also `b` like TeX `\bar`) before the variable name, for tangent direction `d` (like direction or TeX `\dot`).
  
  Beginning from the last step, one should get
  
  * from the residual `0.5*sum((A-Y)**2)` indeed `gA2 = (A2-Y).T`
  * from the last step `A2 = sigmoid(Z2)` you should get `gZ2 = gA2*Dsigmoid(Z2)` (component-wise product)
  * Next up is `Z2 = W2.dot(A1) + b2`. Using generic directions `dA1`, `dW2` (column vectors) etc., the defining relation is
  
    ```
    gZ2 @ dZ2 = gZ2 @ dW2 @ A1 + gZ2 @ W2 @ dA1 + gZ2 @ db2
               = trace(gW2 @ dW2) + gA1 @ dA1 + gb2 * db2
  
    ```
  
    which implies
  
    ```
    gA1 = gZ2 @ W2
    gW2 = A1 @ gZ2 # this is a matrix as product column times row
    gb2 = gZ2
  
    ```
  * next `A1 = relu(Z1)` leads to `gZ1 = gA1 * Drelu(Z1)` (component-wise)
  * finally `Z1 = W1.dot(X) + b1` similar to above
  
    ```
    gX = gZ1 @ W1
    gW1 = X @ gZ1
    gb1 = gZ1
  
    ```
  
  In total, you need some slight modifications in the backward iteration
  
  ```
  def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):
  
      gA2 = (A2 - Y).T
      gZ2 = gA2 * sigmoid_prime(Z2.T)   # Z2*(1-Z2)
  
      gA1 = gZ2 @ W2
      gW2 = A1 @ gZ2
      gb2 = gZ2
  
      gZ1 = gA1 * relu_prime(Z1.T)       # 0.5*(1+signum(Z1))
  
      # gX = gZ1 @ W1
      gW1 = X @ gZ1
      gb1 = gZ1
  
      return gW1.T, gW2.T, gb1.T, gb2.T
  
  ```
  
  Or with the gradients and every equation transposed to above
  
  ```
  def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):
  
      gA2 = A2 - Y
      gZ2 = sigmoid_prime(Z2) * gA2   # Z2*(1-Z2)
  
      gA1 = W2.T @ gZ2
      gW2 = gZ2 @ A1.T
      gb2 = gZ2
  
      gZ1 = relu_prime(Z1) * gA1       # 0.5*(1+signum(Z1))
  
      # gX = W1.T @ gZ1
      gW1 = gZ1 @ X.T
      gb1 = gZ1
  
      return gW1, gW2, gb1, gb2
  
  ```
  
  > **Comments:**
  > **username_0**: I found out I was passing incorrect input data to my forwarrd propagation function. After changing it to `each_example` it did work start to work with increasing accuracy over time. Now I never got accuracy of over .95 with the architecture but I'll take it. However, reading  your comment, I gather I might have  made some errs in back_prop, I'll look into it. I felt pretty confident about it because all the equations came from Ng's Stanford ML lecture. I followed along derived them myself using partial derv.  Only Relu_prime came up during the back_prop in my case.  
  
  ---
  
  