Body: <p>I am developing my ANN <em>from scratch</em> which is supposed to classify <a href="http://yann.lecun.com/exdb/mnist/" rel="nofollow noreferrer">MNIST database</a> of handwritten digits (0-9). My feed-forward fully connected ANN <em>has to be</em> composed of:</p>
  <ol>
  <li>One input layer, with <code>28x28 = 784</code> nodes (that is, features of each image)</li>
  <li>One hidden layer, with any number of neurons (shallow network)</li>
  <li>One output layer, with <code>10</code> nodes (one for each predicted class)</li>
  </ol>
  <p>and <em>has to</em> compute gradient w.r.t. weights and bias thanks to <em>backpropagation</em> algorithm and, finally, it should learn exploiting <em>gradient descent with momentum</em> algorithm.</p>
  <p>The loss function is: <code>cross_entropy</code> on &quot;<code>softmax</code>ed&quot; network's outputs, since the task is about <em>classification</em>.</p>
  <p>Each hidden neuron is activated by the same activation function, I've chosen the <code>sigmoid</code>; meanwhile the output's neurons are activated by the <code>identity</code> function.</p>
  <p>The dataset has been divided into:</p>
  <ol>
  <li><code>60.000</code> training pairs <code>(image, label)</code> - for the training</li>
  <li><code>5000</code> validation pairs <code>(image, label)</code> - for evaluation and select the network which minimize the validation loss</li>
  <li><code>5000</code> testing pairs <code>(image, label)</code> - for testing the model picked using new metrics such as accuracy</li>
  </ol>
  <p>The data has been shuffled invoking <code>sklearn.utils.shuffle</code> method.</p>
  <p>These are my net's performance about training loss, validation loss and validation accuracy:</p>
  <pre><code>E(0) on TrS is: 798288.7537714319  on VS is: 54096.50409967187  Accuracy: 12.1 %
  E(1) on TrS is: 798261.8584179751  on VS is: 54097.23663558976  Accuracy: 12.1 %
  ...
  E(8) on TrS is: 798252.1191081362  on VS is: 54095.5016235736  Accuracy: 12.1 %
  ...
  E(17) on TrS is: 798165.2674011206  on VS is: 54087.2823473459  Accuracy: 12.8 %
  E(18) on TrS is: 798155.0888987815  on VS is: 54086.454077456074  Accuracy: 13.22 %
  ...
  E(32) on TrS is: 798042.8283810444  on VS is: 54076.35518400717  Accuracy: 19.0 %
  E(33) on TrS is: 798033.2512910366  on VS is: 54075.482037626025  Accuracy: 19.36 %
  E(34) on TrS is: 798023.431899881  on VS is: 54074.591145985265  Accuracy: 19.64 %
  E(35) on TrS is: 798013.4023181734  on VS is: 54073.685418577166  Accuracy: 19.759999999999998 %
  E(36) on TrS is: 798003.1960815473  on VS is: 54072.76783050559  Accuracy: 20.080000000000002 %
  ...
  E(47) on TrS is: 797888.8213232228  on VS is: 54062.70342708315  Accuracy: 21.22 %
  E(48) on TrS is: 797879.005388998  on VS is: 54061.854566864626  Accuracy: 21.240000000000002 %
  E(49) on TrS is: 797869.3890292909  on VS is: 54061.02482142968  Accuracy: 21.26 %
  Validation loss is minimum at epoch: 49
  </code></pre>
  <p><a href="https://i.sstatic.net/49TDu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/49TDu.png" alt="training and validation loss" /></a></p>
  <p><a href="https://i.sstatic.net/hP8Di.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hP8Di.png" alt="training loss" /></a></p>
  <p><a href="https://i.sstatic.net/pq1sq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pq1sq.png" alt="validation loss" /></a></p>
  <p>(note that training loss seems to be much higher than validation loss and for this reason they seems to be straight horizontal lines in the first image)</p>
  <p>This is my code:</p>
  <pre><code>import numpy as np
  from scipy.special import expit
  from matplotlib import pyplot as plt
  from mnist.loader import MNIST
  
  
  def relu(a, derivative=False):
      f_a = np.maximum(0, a)
      if derivative:
          return (a &gt; 0) * 1
      return f_a  
  
  def softmax(y):
      e_y = np.exp(y - np.max(y))
      return e_y / e_y.sum()
  
  def cross_entropy(y, t, derivative=False, post_process=True):
      epsilon = 10 ** -308
      if post_process:
          if derivative:
              return y - t
          sm = softmax(y)
          sm = np.clip(sm, epsilon, 1 - epsilon)  # avoids log(0)
          return -np.sum(np.sum(np.multiply(t, np.log(sm)), axis=0))
  
  def sigmoid(a, derivative=False):
      f_a = expit(a)
      if derivative:
          return np.multiply(f_a, (1 - f_a))
      return f_a
  
  def identity(a, derivative=False):
      f_a = a
      if derivative:
          return np.ones(np.shape(a))
      return f_a
  
  def accuracy_score(targets, predictions):
      correct_predictions = 0
      for item in range(np.shape(predictions)[1]):
          argmax_idx = np.argmax(predictions[:, item])
          if targets[argmax_idx, item] == 1:
              correct_predictions += 1
      return correct_predictions / np.shape(predictions)[1]
  
  
  def one_hot(targets):
      return np.asmatrix(np.eye(10)[targets]).T
  
  
  def plot(epochs, loss_train, loss_val):
      plt.plot(epochs, loss_train)
      plt.plot(epochs, loss_val, color=&quot;orange&quot;)
      plt.legend([&quot;Training Loss&quot;, &quot;Validation Loss&quot;])
      plt.xlabel(&quot;Epochs&quot;)
      plt.ylabel(&quot;Loss&quot;)
      plt.grid(True)
      plt.show()
  
  class NeuralNetwork:
  
      def __init__(self):
          self.layers = []
  
      def add_layer(self, layer):
          self.layers.append(layer)
  
      def build(self):
          for i, layer in enumerate(self.layers):
              if i == 0:
                  layer.type = &quot;input&quot;
              else:
                  layer.type = &quot;output&quot; if i == len(self.layers) - 1 else &quot;hidden&quot;
                  layer.configure(self.layers[i - 1].neurons)
  
      def fit(self, X_train, targets_train, X_val, targets_val, max_epochs=50):
          e_loss_train = []
          e_loss_val = []
  
          # Getting the minimum loss on validation set
          predictions_val = self.predict(X_val)
          min_loss_val = cross_entropy(predictions_val, targets_val)
  
          best_net = self  # net which minimize validation loss
          best_epoch = 0  # epoch where the validation loss is minimum
  
          # batch mode
          for epoch in range(max_epochs):
              predictions_train = self.predict(X_train)
              self.back_prop(targets_train, cross_entropy)
              self.learning_rule(l_rate=0.00001, momentum=0.9)
              loss_train = cross_entropy(predictions_train, targets_train)
              e_loss_train.append(loss_train)
  
              # Validation
              predictions_val = self.predict(X_val)
              loss_val = cross_entropy(predictions_val, targets_val)
              e_loss_val.append(loss_val)
  
              print(&quot;E(%d) on TrS is:&quot; % epoch, loss_train, &quot; on VS is:&quot;, loss_val, &quot; Accuracy:&quot;,
                    accuracy_score(targets_val, predictions_val) * 100, &quot;%&quot;)
  
              if loss_val &lt; min_loss_val:
                  min_loss_val = loss_val
                  best_epoch = epoch
                  best_net = self
    
          plot(np.arange(max_epochs), e_loss_train, e_loss_val)
  
          return best_net
  
      # Matrix of predictions where the i-th column corresponds to the i-th item
      def predict(self, dataset):
          z = dataset.T
          for layer in self.layers:
              z = layer.forward_prop_step(z)
          return z
  
      def back_prop(self, target, loss):
          for i, layer in enumerate(self.layers[:0:-1]):
              next_layer = self.layers[-i]
              prev_layer = self.layers[-i - 2]
              layer.back_prop_step(next_layer, prev_layer, target, loss)
  
      def learning_rule(self, l_rate, momentum):
          # Momentum GD
          for layer in [layer for layer in self.layers if layer.type != &quot;input&quot;]:
              layer.update_weights(l_rate, momentum)
              layer.update_bias(l_rate, momentum)
  
  
  class Layer:
  
      def __init__(self, neurons, type=None, activation=None):
          self.dE_dW = None  # derivatives dE/dW where W is the weights matrix
          self.dE_db = None  # derivatives dE/db where b is the bias
          self.dact_a = None  # derivative of the activation function
          self.out = None  # layer output
          self.weights = None  # input weights
          self.bias = None  # layer bias
          self.w_sum = None  # weighted_sum
          self.neurons = neurons  # number of neurons
          self.type = type  # input, hidden or output
          self.activation = activation  # activation function
          self.deltas = None  # for back-prop
  
      def configure(self, prev_layer_neurons):
          self.set_activation()
          self.weights = np.asmatrix(np.random.normal(-0.1, 0.02, (self.neurons, prev_layer_neurons)))
          self.bias = np.asmatrix(np.random.normal(-0.1, 0.02, self.neurons)).T 
  
  
      def set_activation(self):
          if self.activation is None:
              if self.type == &quot;hidden&quot;:
                  self.activation = sigmoid
              elif self.type == &quot;output&quot;:
                  self.activation = identity  # will be softmax in cross entropy calculation
  
      def forward_prop_step(self, z):
          if self.type == &quot;input&quot;:
              self.out = z
          else:
              self.w_sum = np.dot(self.weights, z) + self.bias
              self.out = self.activation(self.w_sum)
          return self.out
  
      def back_prop_step(self, next_layer, prev_layer, target, local_loss):
          if self.type == &quot;output&quot;:
              self.dact_a = self.activation(self.w_sum, derivative=True)
              self.deltas = np.multiply(self.dact_a,
                                        local_loss(self.out, target, derivative=True))
          else:
              self.dact_a = self.activation(self.w_sum, derivative=True)  # (m,batch_size)
              self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weights.T, next_layer.deltas))
  
          self.dE_dW = self.deltas * prev_layer.out.T
  
          self.dE_db = np.sum(self.deltas, axis=1)
  
      def update_weights(self, l_rate, momentum):
          # Momentum GD
          self.weights = self.weights - l_rate * self.dE_dW
          self.weights = -l_rate * self.dE_dW + momentum * self.weights
  
      def update_bias(self, l_rate, momentum):
          # Momentum GD
          self.bias = self.bias - l_rate * self.dE_db
          self.bias = -l_rate * self.dE_db + momentum * self.bias
  
  
  if __name__ == '__main__':
      mndata = MNIST(path=&quot;data&quot;, return_type=&quot;numpy&quot;, mode=&quot;randomly_binarized&quot;)
      X_train, targets_train = mndata.load_training()  # 60.000 images, 28*28 features
      X_val, targets_val = mndata.load_testing()  # 10.000 images, 28*28 features
  
      X_train, targets_train = shuffle(X_train, targets_train.T)
      X_val, targets_val = shuffle(X_val, targets_val.T)
  
      # Getting the test set splitting the validation set in two equal parts
      # Validation set size decreases from 10.000 to 5000 (of course)
      X_val, X_test = np.split(X_val, 2)  # 5000 images, 28*28 features
      targets_val, targets_test = np.split(targets_val, 2)
      X_test, targets_test = shuffle(X_test, targets_test.T)
  
      targets_train = one_hot(targets_train)
      targets_val = one_hot(targets_val)
      targets_test = one_hot(targets_test)
  
      net = NeuralNetwork()
      d = np.shape(X_train)[1]  # number of features, 28*28
      c = np.shape(targets_train)[0]  # number of classes, 10
  
      # Shallow network with 1 hidden neuron
      # That is 784, 1, 10
      for m in (d, 1, c):
          layer = Layer(m)
          net.add_layer(layer)
  
      net.build()
  
      best_net = net.fit(X_train, targets_train, X_val, targets_val, max_epochs=50)
  </code></pre>
  <p>What I have done:</p>
  <ol>
  <li>Set <code>5</code> instead of <code>1</code> hidden neuron</li>
  <li>Add many hidden layers</li>
  <li>Decrease/increase learning rate (<code>l_rate</code>) value</li>
  <li>Decrease/increase <code>momentum</code> (and set it to <code>0</code>)</li>
  <li>Replace <code>sigmoid</code> with <code>relu</code></li>
  </ol>
  <p>but there still is the problem.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-11T23:54:06.190Z
Comments.Text: I wouldn't use epsilon smaller than 1e-10.  Why is this pure-python and not something like Tensorflow or PyTorch?  There is an outstandingly large difference between running this in a compiled and optimized framework that runs on a GPU, and running it hand-coded in an interpreted language that runs on the CPU.  Compiled vs. interpreted I've seen yield 1000x speedup, and cpu compiled vs. gpu compiled i've seen yield a 25x speedup.  My thought: check your error, because I'm not sure you are getting categorical cross-entropy right? https://www.tensorflow.org/datasets/keras_example
Comments.UserDisplayName: EngrStudent
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-12T08:09:23.207Z
Comments.Text: @EngrStudent Because the assignment is about neural network from scratch: no PyTorch and other frameworks.
Comments.UserDisplayName: Anna
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-12T14:49:28.287Z
Comments.Text: The way to do this is build up a step at a time.
  Can you build a classifier that assigns whether a number is positive or negative?  Make a single input, a single hidden node with random weights, and a single sigmoid output.  The output attempts to answer the question "is it positive?".  
  Should be a subset of code you already have, quick to make.  Will let you know that your stuff works.  Then you can work on applying it to larger inputs.  
  Also, you don't submit the package code or results alone, but use them as a sanity check.  If you can't make it work there, you can't make it work.
Comments.UserDisplayName: EngrStudent
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-12T15:01:09.403Z
Comments.Text: which version of 'mnist' are you using because version '0.2.2' in python 3.8 doesn't seem to have "loader".
Comments.UserDisplayName: EngrStudent
Tags: <neural-networks><python><backpropagation><accuracy>
Title: Training loss is decreasing very slowly while learning MNIST database
UserDisplayName: Anna
ThreadText: # Training loss is decreasing very slowly while learning MNIST database
  
  (Asked by: username_0 on 2022-10-11)
  
  I am developing my ANN *from scratch* which is supposed to classify [MNIST database](http://yann.lecun.com/exdb/mnist/) of handwritten digits (0-9). My feed-forward fully connected ANN *has to be* composed of:
  
  1. One input layer, with `28x28 = 784` nodes (that is, features of each image)
  2. One hidden layer, with any number of neurons (shallow network)
  3. One output layer, with `10` nodes (one for each predicted class)
  
  and *has to* compute gradient w.r.t. weights and bias thanks to *backpropagation* algorithm and, finally, it should learn exploiting *gradient descent with momentum* algorithm.
  
  The loss function is: `cross_entropy` on "`softmax`ed" network's outputs, since the task is about *classification*.
  
  Each hidden neuron is activated by the same activation function, I've chosen the `sigmoid`; meanwhile the output's neurons are activated by the `identity` function.
  
  The dataset has been divided into:
  
  1. `60.000` training pairs `(image, label)` - for the training
  2. `5000` validation pairs `(image, label)` - for evaluation and select the network which minimize the validation loss
  3. `5000` testing pairs `(image, label)` - for testing the model picked using new metrics such as accuracy
  
  The data has been shuffled invoking `sklearn.utils.shuffle` method.
  
  These are my net's performance about training loss, validation loss and validation accuracy:
  
  ```
  E(0) on TrS is: 798288.7537714319  on VS is: 54096.50409967187  Accuracy: 12.1 %
  E(1) on TrS is: 798261.8584179751  on VS is: 54097.23663558976  Accuracy: 12.1 %
  ...
  E(8) on TrS is: 798252.1191081362  on VS is: 54095.5016235736  Accuracy: 12.1 %
  ...
  E(17) on TrS is: 798165.2674011206  on VS is: 54087.2823473459  Accuracy: 12.8 %
  E(18) on TrS is: 798155.0888987815  on VS is: 54086.454077456074  Accuracy: 13.22 %
  ...
  E(32) on TrS is: 798042.8283810444  on VS is: 54076.35518400717  Accuracy: 19.0 %
  E(33) on TrS is: 798033.2512910366  on VS is: 54075.482037626025  Accuracy: 19.36 %
  E(34) on TrS is: 798023.431899881  on VS is: 54074.591145985265  Accuracy: 19.64 %
  E(35) on TrS is: 798013.4023181734  on VS is: 54073.685418577166  Accuracy: 19.759999999999998 %
  E(36) on TrS is: 798003.1960815473  on VS is: 54072.76783050559  Accuracy: 20.080000000000002 %
  ...
  E(47) on TrS is: 797888.8213232228  on VS is: 54062.70342708315  Accuracy: 21.22 %
  E(48) on TrS is: 797879.005388998  on VS is: 54061.854566864626  Accuracy: 21.240000000000002 %
  E(49) on TrS is: 797869.3890292909  on VS is: 54061.02482142968  Accuracy: 21.26 %
  Validation loss is minimum at epoch: 49
  
  ```
  
  [![training and validation loss](https://i.sstatic.net/49TDu.png)](https://i.sstatic.net/49TDu.png)
  
  [![training loss](https://i.sstatic.net/hP8Di.png)](https://i.sstatic.net/hP8Di.png)
  
  [![validation loss](https://i.sstatic.net/pq1sq.png)](https://i.sstatic.net/pq1sq.png)
  
  (note that training loss seems to be much higher than validation loss and for this reason they seems to be straight horizontal lines in the first image)
  
  This is my code:
  
  ```
  import numpy as np
  from scipy.special import expit
  from matplotlib import pyplot as plt
  from mnist.loader import MNIST
  
  def relu(a, derivative=False):
      f_a = np.maximum(0, a)
      if derivative:
          return (a > 0) * 1
      return f_a  
  
  def softmax(y):
      e_y = np.exp(y - np.max(y))
      return e_y / e_y.sum()
  
  def cross_entropy(y, t, derivative=False, post_process=True):
      epsilon = 10 ** -308
      if post_process:
          if derivative:
              return y - t
          sm = softmax(y)
          sm = np.clip(sm, epsilon, 1 - epsilon)  # avoids log(0)
          return -np.sum(np.sum(np.multiply(t, np.log(sm)), axis=0))
  
  def sigmoid(a, derivative=False):
      f_a = expit(a)
      if derivative:
          return np.multiply(f_a, (1 - f_a))
      return f_a
  
  def identity(a, derivative=False):
      f_a = a
      if derivative:
          return np.ones(np.shape(a))
      return f_a
  
  def accuracy_score(targets, predictions):
      correct_predictions = 0
      for item in range(np.shape(predictions)[1]):
          argmax_idx = np.argmax(predictions[:, item])
          if targets[argmax_idx, item] == 1:
              correct_predictions += 1
      return correct_predictions / np.shape(predictions)[1]
  
  def one_hot(targets):
      return np.asmatrix(np.eye(10)[targets]).T
  
  def plot(epochs, loss_train, loss_val):
      plt.plot(epochs, loss_train)
      plt.plot(epochs, loss_val, color="orange")
      plt.legend(["Training Loss", "Validation Loss"])
      plt.xlabel("Epochs")
      plt.ylabel("Loss")
      plt.grid(True)
      plt.show()
  
  class NeuralNetwork:
  
      def __init__(self):
          self.layers = []
  
      def add_layer(self, layer):
          self.layers.append(layer)
  
      def build(self):
          for i, layer in enumerate(self.layers):
              if i == 0:
                  layer.type = "input"
              else:
                  layer.type = "output" if i == len(self.layers) - 1 else "hidden"
                  layer.configure(self.layers[i - 1].neurons)
  
      def fit(self, X_train, targets_train, X_val, targets_val, max_epochs=50):
          e_loss_train = []
          e_loss_val = []
  
          # Getting the minimum loss on validation set
          predictions_val = self.predict(X_val)
          min_loss_val = cross_entropy(predictions_val, targets_val)
  
          best_net = self  # net which minimize validation loss
          best_epoch = 0  # epoch where the validation loss is minimum
  
          # batch mode
          for epoch in range(max_epochs):
              predictions_train = self.predict(X_train)
              self.back_prop(targets_train, cross_entropy)
              self.learning_rule(l_rate=0.00001, momentum=0.9)
              loss_train = cross_entropy(predictions_train, targets_train)
              e_loss_train.append(loss_train)
  
              # Validation
              predictions_val = self.predict(X_val)
              loss_val = cross_entropy(predictions_val, targets_val)
              e_loss_val.append(loss_val)
  
              print("E(%d) on TrS is:" % epoch, loss_train, " on VS is:", loss_val, " Accuracy:",
                    accuracy_score(targets_val, predictions_val) * 100, "%")
  
              if loss_val < min_loss_val:
                  min_loss_val = loss_val
                  best_epoch = epoch
                  best_net = self
    
          plot(np.arange(max_epochs), e_loss_train, e_loss_val)
  
          return best_net
  
      # Matrix of predictions where the i-th column corresponds to the i-th item
      def predict(self, dataset):
          z = dataset.T
          for layer in self.layers:
              z = layer.forward_prop_step(z)
          return z
  
      def back_prop(self, target, loss):
          for i, layer in enumerate(self.layers[:0:-1]):
              next_layer = self.layers[-i]
              prev_layer = self.layers[-i - 2]
              layer.back_prop_step(next_layer, prev_layer, target, loss)
  
      def learning_rule(self, l_rate, momentum):
          # Momentum GD
          for layer in [layer for layer in self.layers if layer.type != "input"]:
              layer.update_weights(l_rate, momentum)
              layer.update_bias(l_rate, momentum)
  
  class Layer:
  
      def __init__(self, neurons, type=None, activation=None):
          self.dE_dW = None  # derivatives dE/dW where W is the weights matrix
          self.dE_db = None  # derivatives dE/db where b is the bias
          self.dact_a = None  # derivative of the activation function
          self.out = None  # layer output
          self.weights = None  # input weights
          self.bias = None  # layer bias
          self.w_sum = None  # weighted_sum
          self.neurons = neurons  # number of neurons
          self.type = type  # input, hidden or output
          self.activation = activation  # activation function
          self.deltas = None  # for back-prop
  
      def configure(self, prev_layer_neurons):
          self.set_activation()
          self.weights = np.asmatrix(np.random.normal(-0.1, 0.02, (self.neurons, prev_layer_neurons)))
          self.bias = np.asmatrix(np.random.normal(-0.1, 0.02, self.neurons)).T 
  
      def set_activation(self):
          if self.activation is None:
              if self.type == "hidden":
                  self.activation = sigmoid
              elif self.type == "output":
                  self.activation = identity  # will be softmax in cross entropy calculation
  
      def forward_prop_step(self, z):
          if self.type == "input":
              self.out = z
          else:
              self.w_sum = np.dot(self.weights, z) + self.bias
              self.out = self.activation(self.w_sum)
          return self.out
  
      def back_prop_step(self, next_layer, prev_layer, target, local_loss):
          if self.type == "output":
              self.dact_a = self.activation(self.w_sum, derivative=True)
              self.deltas = np.multiply(self.dact_a,
                                        local_loss(self.out, target, derivative=True))
          else:
              self.dact_a = self.activation(self.w_sum, derivative=True)  # (m,batch_size)
              self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weights.T, next_layer.deltas))
  
          self.dE_dW = self.deltas * prev_layer.out.T
  
          self.dE_db = np.sum(self.deltas, axis=1)
  
      def update_weights(self, l_rate, momentum):
          # Momentum GD
          self.weights = self.weights - l_rate * self.dE_dW
          self.weights = -l_rate * self.dE_dW + momentum * self.weights
  
      def update_bias(self, l_rate, momentum):
          # Momentum GD
          self.bias = self.bias - l_rate * self.dE_db
          self.bias = -l_rate * self.dE_db + momentum * self.bias
  
  if __name__ == '__main__':
      mndata = MNIST(path="data", return_type="numpy", mode="randomly_binarized")
      X_train, targets_train = mndata.load_training()  # 60.000 images, 28*28 features
      X_val, targets_val = mndata.load_testing()  # 10.000 images, 28*28 features
  
      X_train, targets_train = shuffle(X_train, targets_train.T)
      X_val, targets_val = shuffle(X_val, targets_val.T)
  
      # Getting the test set splitting the validation set in two equal parts
      # Validation set size decreases from 10.000 to 5000 (of course)
      X_val, X_test = np.split(X_val, 2)  # 5000 images, 28*28 features
      targets_val, targets_test = np.split(targets_val, 2)
      X_test, targets_test = shuffle(X_test, targets_test.T)
  
      targets_train = one_hot(targets_train)
      targets_val = one_hot(targets_val)
      targets_test = one_hot(targets_test)
  
      net = NeuralNetwork()
      d = np.shape(X_train)[1]  # number of features, 28*28
      c = np.shape(targets_train)[0]  # number of classes, 10
  
      # Shallow network with 1 hidden neuron
      # That is 784, 1, 10
      for m in (d, 1, c):
          layer = Layer(m)
          net.add_layer(layer)
  
      net.build()
  
      best_net = net.fit(X_train, targets_train, X_val, targets_val, max_epochs=50)
  
  ```
  
  What I have done:
  
  1. Set `5` instead of `1` hidden neuron
  2. Add many hidden layers
  3. Decrease/increase learning rate (`l_rate`) value
  4. Decrease/increase `momentum` (and set it to `0`)
  5. Replace `sigmoid` with `relu`
  
  but there still is the problem.
  
  **Tags:** neural-networks, python, backpropagation, accuracy
  
  > **Comments:**
  > **username_1**: I wouldn't use epsilon smaller than 1e-10.  Why is this pure-python and not something like Tensorflow or PyTorch?  There is an outstandingly large difference between running this in a compiled and optimized framework that runs on a GPU, and running it hand-coded in an interpreted language that runs on the CPU.  Compiled vs. interpreted I've seen yield 1000x speedup, and cpu compiled vs. gpu compiled i've seen yield a 25x speedup.  My thought: check your error, because I'm not sure you are getting categorical cross-entropy right? https://www.tensorflow.org/datasets/keras_example  
  > **username_0**: @username_1 Because the assignment is about neural network from scratch: no PyTorch and other frameworks.  
  > **username_1**: The way to do this is build up a step at a time. Can you build a classifier that assigns whether a number is positive or negative?  Make a single input, a single hidden node with random weights, and a single sigmoid output.  The output attempts to answer the question "is it positive?".   Should be a subset of code you already have, quick to make.  Will let you know that your stuff works.  Then you can work on applying it to larger inputs.   Also, you don't submit the package code or results alone, but use them as a sanity check.  If you can't make it work there, you can't make it work.  
  > **username_1**: which version of 'mnist' are you using because version '0.2.2' in python 3.8 doesn't seem to have "loader".  
  
  ---
  
  ## 0 answers
  
  