Body: <p>I am developing my ANN <em>from scratch</em> which is supposed to classify <a href="http://yann.lecun.com/exdb/mnist/" rel="nofollow noreferrer">MNIST database</a> of handwritten digits (0-9). My feed-forward fully connected ANN <em>has to be</em> composed of:</p>
  <ol>
  <li>One input layer, with <code>28x28 = 784</code> nodes (that is, features of each image)</li>
  <li>One hidden layer, with <code>1,2,3,4</code> or <code>5</code> nodes (shallow network)</li>
  <li>One output layer, with <code>10</code> nodes (one for each predicted class)</li>
  </ol>
  <p>and <em>has to</em> compute gradient w.r.t. weights and bias thanks to backpropagation algorithm and, finally, it should learn exploiting gradient descent with momentum algorithm.</p>
  <p>The loss function is: <code>cross-entropy + softmax</code>, since the task is about <em>classification</em>.</p>
  <p>Each hidden neuron is activated by the same activation function, the <code>sigmoid</code>; meanwhile the output's neurons are activated by the <code>identity</code> function.</p>
  <p>The dataset has been divided into:</p>
  <ol>
  <li><code>60.000</code> training pairs <code>(image, label)</code> - for the training</li>
  <li><code>5000</code> validation pairs <code>(image, label)</code> - for evaluation and select the network which minimize the validation loss</li>
  <li><code>5000</code> testing pairs <code>(image, label)</code> - for testing the model picked using new metrics such as accuracy</li>
  </ol>
  <p>This is the curves I get about training and validation loss: <a href="https://i.sstatic.net/tkCf3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tkCf3.png" alt="training-validation loss" /></a> and these are some logs about their loss and accuracy on the validation set:</p>
  <pre><code>E(0) on TrS is: -0.09995663952048875  on VS is: -0.10000215508367913  Accuracy: 9.120000000000001 %
  E(1) on TrS is: -0.09997782058505714  on VS is: -0.10000721545529842  Accuracy: 9.120000000000001 %
  ...
  E(47) on TrS is: -0.1000167196491167  on VS is: -0.10001587893489938  Accuracy: 11.42 %
  E(48) on TrS is: -0.10001676221437443  on VS is: -0.10001594086326895  Accuracy: 11.42 %
  E(49) on TrS is: -0.10001680503178402  on VS is: -0.10001599995199295  Accuracy: 11.42 %
  
  Validation loss is minimum at epoch: 22
  </code></pre>
  <p>Finally, if I test the network at epoch <code>22</code>, which is the epoch where the validation loss is minimum, I get the following result:</p>
  <pre><code>Accuracy score on test set is: 11.28 %
  </code></pre>
  <p>These are my questions: as you can see, the accuracy remains (almost) constant and it is <em>very</em> low. It has a low value on the test set as well and on the training set. Is that normal? If it is <em>not</em>, how can I fix?</p>
  <p>This is my code:</p>
  <pre><code>import numpy as np
  from scipy.special import xlog1py
  from scipy.special import expit
  from matplotlib import pyplot as plt
  from mnist.loader import MNIST
  
  
  def softmax(y):
      e_y = np.exp(y - np.max(y))
      return e_y / e_y.sum()
  
  def cross_entropy(y, t, derivative=False, post_process=True):
      if post_process:
          if derivative:
              return y - t
          return -np.sum(np.sum(xlog1py(t, softmax(y)), axis=0))
  
  def sigmoid(a, derivative=False):
      f_a = expit(-a)
      if derivative:
          return np.multiply(f_a, (1 - f_a))
      return f_a
  
  def identity(a, derivative=False):
      f_a = a
      if derivative:
          return np.ones(np.shape(a))
      return f_a
  
  def accuracy_score(targets, predictions):
      correct_predictions = 0
      for item in range(np.shape(predictions)[1]):
          # print(predictions[:, item])  # dbg
          argmax_idx = np.argmax(predictions[:, item])
          # print(&quot;argmax idx&quot;, argmax_idx)  # dbg
          # print(targets_test[:, item])  # dbg
          if targets[argmax_idx, item] == 1:
              correct_predictions += 1
      return correct_predictions / np.shape(predictions)[1]
  
  
  def one_hot(targets):
      return np.asmatrix(np.eye(np.max(targets) + 1)[targets]).T
  
  
  def plot(epochs, loss_train, loss_val):
      plt.plot(epochs, loss_train)
      plt.plot(epochs, loss_val, color=&quot;orange&quot;)
      plt.legend([&quot;Training Loss&quot;, &quot;Validation Loss&quot;])
      plt.xlabel(&quot;Epochs&quot;)
      plt.ylabel(&quot;Loss&quot;)
      plt.grid(True)
      plt.show()
  
  class NeuralNetwork:
  
      def __init__(self):
          self.layers = []
  
      def add_layer(self, layer):
          self.layers.append(layer)
  
      def build(self):
          for i, layer in enumerate(self.layers):
              if i == 0:
                  layer.type = &quot;input&quot;
              else:
                  layer.type = &quot;output&quot; if i == len(self.layers) - 1 else &quot;hidden&quot;
                  layer.configure(self.layers[i - 1].neurons)
  
      def fit(self, X_train, targets_train, X_val, targets_val, max_epochs=50):
          e_loss_train = []
          e_loss_val = []
  
          # Getting the minimum loss on validation set
          predictions_val = self.predict(X_val)
          min_loss_val = cross_entropy(predictions_val, targets_val)
  
          best_net = self  # net which minimize validation loss
          best_epoch = 0  # epoch where the validation loss is minimum
  
          # batch mode
          for epoch in range(max_epochs):
              predictions_train = self.predict(X_train)
              self.back_prop(targets_train, cross_entropy)
              self.learning_rule(l_rate=0.000001, momentum=0.9)
              loss_train = cross_entropy(predictions_train, targets_train)
              e_loss_train.append(loss_train)
  
              # Validation
              predictions_val = self.predict(X_val)
              loss_val = cross_entropy(predictions_val, targets_val)
              e_loss_val.append(loss_val)
  
              print(&quot;E(%d) on TrS is:&quot; % epoch, loss_train, &quot; on VS is:&quot;, loss_val, &quot; Accuracy:&quot;,
                    accuracy_score(targets_val, predictions_val) * 100, &quot;%&quot;)
  
              if loss_val &lt; min_loss_val:
                  min_loss_val = loss_val
                  best_epoch = epoch
                  best_net = self
  
          print(&quot;Validation loss is minimum at epoch:&quot;, best_epoch)
  
          plot(np.arange(max_epochs), e_loss_train, e_loss_val)
  
          return best_net
  
      # Matrix of predictions where the i-th column corresponds to the i-th item
      def predict(self, dataset):
          z = dataset.T
          for layer in self.layers:
              z = layer.forward_prop_step(z)
          return z
  
      def back_prop(self, target, loss):
          for i, layer in enumerate(self.layers[:0:-1]):
              next_layer = self.layers[-i]
              prev_layer = self.layers[-i - 2]
              layer.back_prop_step(next_layer, prev_layer, target, loss)
  
      def learning_rule(self, l_rate, momentum):
          # Momentum GD
          for layer in [layer for layer in self.layers if layer.type != &quot;input&quot;]:
              layer.update_weights(l_rate, momentum)
              layer.update_bias(l_rate, momentum)
  
  
  class Layer:
  
      def __init__(self, neurons, type=None, activation=None):
          self.dE_dW = None  # derivatives dE/dW where W is the weights matrix
          self.dE_db = None  # derivatives dE/db where b is the bias
          self.dact_a = None  # derivative of the activation function
          self.out = None  # layer output
          self.weights = None  # input weights
          self.bias = None  # layer bias
          self.w_sum = None  # weighted_sum
          self.neurons = neurons  # number of neurons
          self.type = type  # input, hidden or output
          self.activation = activation  # activation function
          self.deltas = None  # for back-prop
  
      def configure(self, prev_layer_neurons):
          self.set_activation()
          self.weights = np.asmatrix(np.random.normal(-0.1, 0.02, (self.neurons, prev_layer_neurons)))
          self.bias = np.asmatrix(np.random.normal(-0.1, 0.02, self.neurons)).T  # vettore colonna
  
  
      def set_activation(self):
          if self.activation is None:
              if self.type == &quot;hidden&quot;:
                  self.activation = sigmoid
              elif self.type == &quot;output&quot;:
                  self.activation = identity
  
      def forward_prop_step(self, z):
          if self.type == &quot;input&quot;:
              self.out = z
          else:
              self.w_sum = np.dot(self.weights, z) + self.bias
              self.out = self.activation(self.w_sum)
          return self.out
  
      def back_prop_step(self, next_layer, prev_layer, target, local_loss):
          if self.type == &quot;output&quot;:
              self.dact_a = self.activation(self.w_sum, derivative=True)
              self.deltas = np.multiply(self.dact_a,
                                        local_loss(self.out, target, derivative=True))
          else:
              self.dact_a = self.activation(self.w_sum, derivative=True)  # (m,batch_size)
              self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weights.T, next_layer.deltas))
  
          self.dE_dW = self.deltas * prev_layer.out.T
  
          self.dE_db = np.sum(self.deltas, axis=1)
  
      def update_weights(self, l_rate, momentum):
          # Momentum GD
          self.weights = self.weights - l_rate * self.dE_dW
          self.weights = -l_rate * self.dE_dW + momentum * self.weights
  
      def update_bias(self, l_rate, momentum):
          # Momentum GD
          self.bias = self.bias - l_rate * self.dE_db
          self.bias = -l_rate * self.dE_db + momentum * self.bias
  
  
  if __name__ == '__main__':
      mndata = MNIST(path=&quot;data&quot;, return_type=&quot;numpy&quot;, mode=&quot;randomly_binarized&quot;)
      X_train, targets_train = mndata.load_training()  # 60.000 images, 28*28 features
      X_val, targets_val = mndata.load_testing()  # 10.000 images, 28*28 features
  
      # Getting the test set splitting the validation set in two equal parts
      # Validation set size decreases from 10.000 to 5000 (of course)
      X_val, X_test = np.split(X_val, 2)  # 5000 images, 28*28 features
      targets_val, targets_test = np.split(targets_val, 2)
  
      targets_train = one_hot(targets_train)
      targets_val = one_hot(targets_val)
      targets_test = one_hot(targets_test)
  
      net = NeuralNetwork()
      d = np.shape(X_train)[1]  # number of features, 28*28
      c = np.shape(targets_train)[0]  # number of classes, 10
  
      # Shallow network with 1 hidden neuron
      # That is 784, 1, 10
      for m in (d, 1, c):
          layer = Layer(m)
          net.add_layer(layer)
  
      net.build()
  
      best_net = net.fit(X_train, targets_train, X_val, targets_val, max_epochs=50)
  
      predictions = best_net.predict(X_test)
      print(&quot;Accuracy score on test set is:&quot;, accuracy_score(targets_test, predictions) * 100, &quot;%&quot;)
  </code></pre>
  <p>Note that:</p>
  <ol>
  <li>The performance are similar even though I set <code>5</code> hidden neurons
  instead of <code>1</code></li>
  <li>Accuracy on training set is very low as well (around <code>11 %</code>)</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-11T09:31:44.070Z
Comments.Text: 5 nodes is basically nothing, have you tried with 500? If the accuracy remain the same then I would start worrying about the code implementation.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-11T10:49:00.297Z
Comments.Text: @EdoardoGuerriero accuracy is low
Comments.UserDisplayName: tail
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-11T15:49:00.183Z
Comments.Text: @EdoardoGuerriero I would disagree with the fact that the number of nodes is not enough. According to the [original mnist paper](http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf) a naiive linear classifier got an *error rate* of roughly 12%. Note that error rate is different than accuracy. Also if accuracy is 11%, then that means it's basically picking a number at random.
Comments.UserDisplayName: juicedatom
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-11T16:10:11.480Z
Comments.Text: Any solution? @juicedatom
Comments.UserDisplayName: tail
Tags: <neural-networks><machine-learning><python><backpropagation><accuracy>
Title: Very low accuracy (0.11) and it remains constant after few epochs on MNIST database
UserDisplayName: tail
ThreadText: # Very low accuracy (0.11) and it remains constant after few epochs on MNIST database
  
  (Asked by: username_0 on 2022-10-09)
  
  I am developing my ANN from scratch which is supposed to classify MNIST database of handwritten digits (0-9). My feed-forward fully connected ANN has to be composed of:
  
  One input layer, with 28x28 = 784 nodes (that is, features of each image)
  One hidden layer, with 1,2,3,4 or 5 nodes (shallow network)
  One output layer, with 10 nodes (one for each predicted class)
  
  and has to compute gradient w.r.t. weights and bias thanks to backpropagation algorithm and, finally, it should learn exploiting gradient descent with momentum algorithm.
  The loss function is: cross-entropy + softmax, since the task is about classification.
  Each hidden neuron is activated by the same activation function, the sigmoid; meanwhile the output's neurons are activated by the identity function.
  The dataset has been divided into:
  
  60.000 training pairs (image, label) - for the training
  5000 validation pairs (image, label) - for evaluation and select the network which minimize the validation loss
  5000 testing pairs (image, label) - for testing the model picked using new metrics such as accuracy
  
  This is the curves I get about training and validation loss:  and these are some logs about their loss and accuracy on the validation set:
  E(0) on TrS is: -0.09995663952048875  on VS is: -0.10000215508367913  Accuracy: 9.120000000000001 %
  E(1) on TrS is: -0.09997782058505714  on VS is: -0.10000721545529842  Accuracy: 9.120000000000001 %
  ...
  E(47) on TrS is: -0.1000167196491167  on VS is: -0.10001587893489938  Accuracy: 11.42 %
  E(48) on TrS is: -0.10001676221437443  on VS is: -0.10001594086326895  Accuracy: 11.42 %
  E(49) on TrS is: -0.10001680503178402  on VS is: -0.10001599995199295  Accuracy: 11.42 %
  
  Validation loss is minimum at epoch: 22
  
  Finally, if I test the network at epoch 22, which is the epoch where the validation loss is minimum, I get the following result:
  Accuracy score on test set is: 11.28 %
  
  These are my questions: as you can see, the accuracy remains (almost) constant and it is very low. It has a low value on the test set as well and on the training set. Is that normal? If it is not, how can I fix?
  This is my code:
  import numpy as np
  from scipy.special import xlog1py
  from scipy.special import expit
  from matplotlib import pyplot as plt
  from mnist.loader import MNIST
  
  
  def softmax(y):
      e_y = np.exp(y - np.max(y))
      return e_y / e_y.sum()
  
  def cross_entropy(y, t, derivative=False, post_process=True):
      if post_process:
          if derivative:
              return y - t
          return -np.sum(np.sum(xlog1py(t, softmax(y)), axis=0))
  
  def sigmoid(a, derivative=False):
      f_a = expit(-a)
      if derivative:
          return np.multiply(f_a, (1 - f_a))
      return f_a
  
  def identity(a, derivative=False):
      f_a = a
      if derivative:
          return np.ones(np.shape(a))
      return f_a
  
  def accuracy_score(targets, predictions):
      correct_predictions = 0
      for item in range(np.shape(predictions)[1]):
          # print(predictions[:, item])  # dbg
          argmax_idx = np.argmax(predictions[:, item])
          # print("argmax idx", argmax_idx)  # dbg
          # print(targets_test[:, item])  # dbg
          if targets[argmax_idx, item] == 1:
              correct_predictions += 1
      return correct_predictions / np.shape(predictions)[1]
  
  
  def one_hot(targets):
      return np.asmatrix(np.eye(np.max(targets) + 1)[targets]).T
  
  
  def plot(epochs, loss_train, loss_val):
      plt.plot(epochs, loss_train)
      plt.plot(epochs, loss_val, color="orange")
      plt.legend(["Training Loss", "Validation Loss"])
      plt.xlabel("Epochs")
      plt.ylabel("Loss")
      plt.grid(True)
      plt.show()
  
  class NeuralNetwork:
  
      def __init__(self):
          self.layers = []
  
      def add_layer(self, layer):
          self.layers.append(layer)
  
      def build(self):
          for i, layer in enumerate(self.layers):
              if i == 0:
                  layer.type = "input"
              else:
                  layer.type = "output" if i == len(self.layers) - 1 else "hidden"
                  layer.configure(self.layers[i - 1].neurons)
  
      def fit(self, X_train, targets_train, X_val, targets_val, max_epochs=50):
          e_loss_train = []
          e_loss_val = []
  
          # Getting the minimum loss on validation set
          predictions_val = self.predict(X_val)
          min_loss_val = cross_entropy(predictions_val, targets_val)
  
          best_net = self  # net which minimize validation loss
          best_epoch = 0  # epoch where the validation loss is minimum
  
          # batch mode
          for epoch in range(max_epochs):
              predictions_train = self.predict(X_train)
              self.back_prop(targets_train, cross_entropy)
              self.learning_rule(l_rate=0.000001, momentum=0.9)
              loss_train = cross_entropy(predictions_train, targets_train)
              e_loss_train.append(loss_train)
  
              # Validation
              predictions_val = self.predict(X_val)
              loss_val = cross_entropy(predictions_val, targets_val)
              e_loss_val.append(loss_val)
  
              print("E(%d) on TrS is:" % epoch, loss_train, " on VS is:", loss_val, " Accuracy:",
                    accuracy_score(targets_val, predictions_val) * 100, "%")
  
              if loss_val < min_loss_val:
                  min_loss_val = loss_val
                  best_epoch = epoch
                  best_net = self
  
          print("Validation loss is minimum at epoch:", best_epoch)
  
          plot(np.arange(max_epochs), e_loss_train, e_loss_val)
  
          return best_net
  
      # Matrix of predictions where the i-th column corresponds to the i-th item
      def predict(self, dataset):
          z = dataset.T
          for layer in self.layers:
              z = layer.forward_prop_step(z)
          return z
  
      def back_prop(self, target, loss):
          for i, layer in enumerate(self.layers[:0:-1]):
              next_layer = self.layers[-i]
              prev_layer = self.layers[-i - 2]
              layer.back_prop_step(next_layer, prev_layer, target, loss)
  
      def learning_rule(self, l_rate, momentum):
          # Momentum GD
          for layer in [layer for layer in self.layers if layer.type != "input"]:
              layer.update_weights(l_rate, momentum)
              layer.update_bias(l_rate, momentum)
  
  
  class Layer:
  
      def __init__(self, neurons, type=None, activation=None):
          self.dE_dW = None  # derivatives dE/dW where W is the weights matrix
          self.dE_db = None  # derivatives dE/db where b is the bias
          self.dact_a = None  # derivative of the activation function
          self.out = None  # layer output
          self.weights = None  # input weights
          self.bias = None  # layer bias
          self.w_sum = None  # weighted_sum
          self.neurons = neurons  # number of neurons
          self.type = type  # input, hidden or output
          self.activation = activation  # activation function
          self.deltas = None  # for back-prop
  
      def configure(self, prev_layer_neurons):
          self.set_activation()
          self.weights = np.asmatrix(np.random.normal(-0.1, 0.02, (self.neurons, prev_layer_neurons)))
          self.bias = np.asmatrix(np.random.normal(-0.1, 0.02, self.neurons)).T  # vettore colonna
  
  
      def set_activation(self):
          if self.activation is None:
              if self.type == "hidden":
                  self.activation = sigmoid
              elif self.type == "output":
                  self.activation = identity
  
      def forward_prop_step(self, z):
          if self.type == "input":
              self.out = z
          else:
              self.w_sum = np.dot(self.weights, z) + self.bias
              self.out = self.activation(self.w_sum)
          return self.out
  
      def back_prop_step(self, next_layer, prev_layer, target, local_loss):
          if self.type == "output":
              self.dact_a = self.activation(self.w_sum, derivative=True)
              self.deltas = np.multiply(self.dact_a,
                                        local_loss(self.out, target, derivative=True))
          else:
              self.dact_a = self.activation(self.w_sum, derivative=True)  # (m,batch_size)
              self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weights.T, next_layer.deltas))
  
          self.dE_dW = self.deltas * prev_layer.out.T
  
          self.dE_db = np.sum(self.deltas, axis=1)
  
      def update_weights(self, l_rate, momentum):
          # Momentum GD
          self.weights = self.weights - l_rate * self.dE_dW
          self.weights = -l_rate * self.dE_dW + momentum * self.weights
  
      def update_bias(self, l_rate, momentum):
          # Momentum GD
          self.bias = self.bias - l_rate * self.dE_db
          self.bias = -l_rate * self.dE_db + momentum * self.bias
  
  
  if __name__ == '__main__':
      mndata = MNIST(path="data", return_type="numpy", mode="randomly_binarized")
      X_train, targets_train = mndata.load_training()  # 60.000 images, 28*28 features
      X_val, targets_val = mndata.load_testing()  # 10.000 images, 28*28 features
  
      # Getting the test set splitting the validation set in two equal parts
      # Validation set size decreases from 10.000 to 5000 (of course)
      X_val, X_test = np.split(X_val, 2)  # 5000 images, 28*28 features
      targets_val, targets_test = np.split(targets_val, 2)
  
      targets_train = one_hot(targets_train)
      targets_val = one_hot(targets_val)
      targets_test = one_hot(targets_test)
  
      net = NeuralNetwork()
      d = np.shape(X_train)[1]  # number of features, 28*28
      c = np.shape(targets_train)[0]  # number of classes, 10
  
      # Shallow network with 1 hidden neuron
      # That is 784, 1, 10
      for m in (d, 1, c):
          layer = Layer(m)
          net.add_layer(layer)
  
      net.build()
  
      best_net = net.fit(X_train, targets_train, X_val, targets_val, max_epochs=50)
  
      predictions = best_net.predict(X_test)
      print("Accuracy score on test set is:", accuracy_score(targets_test, predictions) * 100, "%")
  
  Note that:
  
  The performance are similar even though I set 5 hidden neurons
  instead of 1
  Accuracy on training set is very low as well (around 11 %)
  
  **Tags:** neural-networks, machine-learning, python, backpropagation, accuracy
  > **Comments:**
  > **username_1**: 5 nodes is basically nothing, have you tried with 500? If the accuracy remain the same then I would start worrying about the code implementation.  
  > **username_0**: @EdoardoGuerriero accuracy is low  
  > **username_2**: @EdoardoGuerriero I would disagree with the fact that the number of nodes is not enough. According to the [original mnist paper](http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf) a naiive linear classifier got an *error rate* of roughly 12%. Note that error rate is different than accuracy. Also if accuracy is 11%, then that means it's basically picking a number at random.  
  > **username_0**: Any solution? @juicedatom  
  
  ---
  
  ## 0 answers
  
  