Body: <p>I am developing my ANN <em>from scratch</em> which is supposed to classify <a href="http://yann.lecun.com/exdb/mnist/" rel="nofollow noreferrer">MNIST database</a> of handwritten digits (0-9). My feed-forward fully connected ANN <em>has to be</em> composed of:</p>
  <ol>
  <li>One input layer, with <code>28x28 = 784</code> nodes (that is, features of each image)</li>
  <li>One hidden layer, with <code>1,2,3,4</code> or <code>5</code> nodes (shallow network)</li>
  <li>One output layer, with <code>10</code> nodes (one for each predicted class)</li>
  </ol>
  <p>and <em>has to</em> compute gradient w.r.t. weights and bias thanks to backpropagation algorithm and, finally, it should learn exploiting gradient descent with momentum algorithm.</p>
  <p>The loss function is: <code>cross-entropy + softmax</code>, since the task is about <em>classification</em>.</p>
  <p>Each hidden neuron is activated by the same activation function, the <code>sigmoid</code>; meanwhile the output's neurons are activated by the <code>identity</code> function.</p>
  <p>The dataset has been divided into:</p>
  <ol>
  <li><code>60.000</code> training pairs <code>(image, label)</code> - for the training</li>
  <li><code>5000</code> validation pairs <code>(image, label)</code> - for evaluation and select the network which minimize the validation loss</li>
  <li><code>5000</code> testing pairs <code>(image, label)</code> - for testing the model picked using new metrics such as accuracy</li>
  </ol>
  <p>This is the curves I get about training and validation loss: <a href="https://i.sstatic.net/tkCf3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tkCf3.png" alt="training-validation loss" /></a> and these are some logs about their loss and accuracy on the validation set:</p>
  <pre><code>E(0) on TrS is: -0.09995663952048875  on VS is: -0.10000215508367913  Accuracy: 9.120000000000001 %
  E(1) on TrS is: -0.09997782058505714  on VS is: -0.10000721545529842  Accuracy: 9.120000000000001 %
  ...
  E(47) on TrS is: -0.1000167196491167  on VS is: -0.10001587893489938  Accuracy: 11.42 %
  E(48) on TrS is: -0.10001676221437443  on VS is: -0.10001594086326895  Accuracy: 11.42 %
  E(49) on TrS is: -0.10001680503178402  on VS is: -0.10001599995199295  Accuracy: 11.42 %
  
  Validation loss is minimum at epoch: 22
  </code></pre>
  <p>Finally, if I test the network at epoch <code>22</code>, which is the epoch where the validation loss is minimum, I get the following result:</p>
  <pre><code>Accuracy score on test set is: 11.28 %
  </code></pre>
  <p>These are my questions: as you can see, the accuracy remains (almost) constant and it is <em>very</em> low. It has a low value on the test set as well and on the training set. Is that normal? If it is <em>not</em>, how can I fix?</p>
  <p>This is my code:</p>
  <pre><code>import numpy as np
  from scipy.special import xlog1py
  from scipy.special import expit
  from matplotlib import pyplot as plt
  from mnist.loader import MNIST
  
  
  def softmax(y):
      e_y = np.exp(y - np.max(y))
      return e_y / e_y.sum()
  
  def cross_entropy(y, t, derivative=False, post_process=True):
      if post_process:
          if derivative:
              return y - t
          return -np.sum(np.sum(xlog1py(t, softmax(y)), axis=0))
  
  def sigmoid(a, derivative=False):
      f_a = expit(-a)
      if derivative:
          return np.multiply(f_a, (1 - f_a))
      return f_a
  
  def identity(a, derivative=False):
      f_a = a
      if derivative:
          return np.ones(np.shape(a))
      return f_a
  
  def accuracy_score(targets, predictions):
      correct_predictions = 0
      for item in range(np.shape(predictions)[1]):
          # print(predictions[:, item])  # dbg
          argmax_idx = np.argmax(predictions[:, item])
          # print(&quot;argmax idx&quot;, argmax_idx)  # dbg
          # print(targets_test[:, item])  # dbg
          if targets[argmax_idx, item] == 1:
              correct_predictions += 1
      return correct_predictions / np.shape(predictions)[1]
  
  
  def one_hot(targets):
      return np.asmatrix(np.eye(np.max(targets) + 1)[targets]).T
  
  
  def plot(epochs, loss_train, loss_val):
      plt.plot(epochs, loss_train)
      plt.plot(epochs, loss_val, color=&quot;orange&quot;)
      plt.legend([&quot;Training Loss&quot;, &quot;Validation Loss&quot;])
      plt.xlabel(&quot;Epochs&quot;)
      plt.ylabel(&quot;Loss&quot;)
      plt.grid(True)
      plt.show()
  
  class NeuralNetwork:
  
      def __init__(self):
          self.layers = []
  
      def add_layer(self, layer):
          self.layers.append(layer)
  
      def build(self):
          for i, layer in enumerate(self.layers):
              if i == 0:
                  layer.type = &quot;input&quot;
              else:
                  layer.type = &quot;output&quot; if i == len(self.layers) - 1 else &quot;hidden&quot;
                  layer.configure(self.layers[i - 1].neurons)
  
      def fit(self, X_train, targets_train, X_val, targets_val, max_epochs=50):
          e_loss_train = []
          e_loss_val = []
  
          # Getting the minimum loss on validation set
          predictions_val = self.predict(X_val)
          min_loss_val = cross_entropy(predictions_val, targets_val)
  
          best_net = self  # net which minimize validation loss
          best_epoch = 0  # epoch where the validation loss is minimum
  
          # batch mode
          for epoch in range(max_epochs):
              predictions_train = self.predict(X_train)
              self.back_prop(targets_train, cross_entropy)
              self.learning_rule(l_rate=0.000001, momentum=0.9)
              loss_train = cross_entropy(predictions_train, targets_train)
              e_loss_train.append(loss_train)
  
              # Validation
              predictions_val = self.predict(X_val)
              loss_val = cross_entropy(predictions_val, targets_val)
              e_loss_val.append(loss_val)
  
              print(&quot;E(%d) on TrS is:&quot; % epoch, loss_train, &quot; on VS is:&quot;, loss_val, &quot; Accuracy:&quot;,
                    accuracy_score(targets_val, predictions_val) * 100, &quot;%&quot;)
  
              if loss_val &lt; min_loss_val:
                  min_loss_val = loss_val
                  best_epoch = epoch
                  best_net = self
  
          print(&quot;Validation loss is minimum at epoch:&quot;, best_epoch)
  
          plot(np.arange(max_epochs), e_loss_train, e_loss_val)
  
          return best_net
  
      # Matrix of predictions where the i-th column corresponds to the i-th item
      def predict(self, dataset):
          z = dataset.T
          for layer in self.layers:
              z = layer.forward_prop_step(z)
          return z
  
      def back_prop(self, target, loss):
          for i, layer in enumerate(self.layers[:0:-1]):
              next_layer = self.layers[-i]
              prev_layer = self.layers[-i - 2]
              layer.back_prop_step(next_layer, prev_layer, target, loss)
  
      def learning_rule(self, l_rate, momentum):
          # Momentum GD
          for layer in [layer for layer in self.layers if layer.type != &quot;input&quot;]:
              layer.update_weights(l_rate, momentum)
              layer.update_bias(l_rate, momentum)
  
  
  class Layer:
  
      def __init__(self, neurons, type=None, activation=None):
          self.dE_dW = None  # derivatives dE/dW where W is the weights matrix
          self.dE_db = None  # derivatives dE/db where b is the bias
          self.dact_a = None  # derivative of the activation function
          self.out = None  # layer output
          self.weights = None  # input weights
          self.bias = None  # layer bias
          self.w_sum = None  # weighted_sum
          self.neurons = neurons  # number of neurons
          self.type = type  # input, hidden or output
          self.activation = activation  # activation function
          self.deltas = None  # for back-prop
  
      def configure(self, prev_layer_neurons):
          self.set_activation()
          self.weights = np.asmatrix(np.random.normal(-0.1, 0.02, (self.neurons, prev_layer_neurons)))
          self.bias = np.asmatrix(np.random.normal(-0.1, 0.02, self.neurons)).T  # vettore colonna
  
  
      def set_activation(self):
          if self.activation is None:
              if self.type == &quot;hidden&quot;:
                  self.activation = sigmoid
              elif self.type == &quot;output&quot;:
                  self.activation = identity
  
      def forward_prop_step(self, z):
          if self.type == &quot;input&quot;:
              self.out = z
          else:
              self.w_sum = np.dot(self.weights, z) + self.bias
              self.out = self.activation(self.w_sum)
          return self.out
  
      def back_prop_step(self, next_layer, prev_layer, target, local_loss):
          if self.type == &quot;output&quot;:
              self.dact_a = self.activation(self.w_sum, derivative=True)
              self.deltas = np.multiply(self.dact_a,
                                        local_loss(self.out, target, derivative=True))
          else:
              self.dact_a = self.activation(self.w_sum, derivative=True)  # (m,batch_size)
              self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weights.T, next_layer.deltas))
  
          self.dE_dW = self.deltas * prev_layer.out.T
  
          self.dE_db = np.sum(self.deltas, axis=1)
  
      def update_weights(self, l_rate, momentum):
          # Momentum GD
          self.weights = self.weights - l_rate * self.dE_dW
          self.weights = -l_rate * self.dE_dW + momentum * self.weights
  
      def update_bias(self, l_rate, momentum):
          # Momentum GD
          self.bias = self.bias - l_rate * self.dE_db
          self.bias = -l_rate * self.dE_db + momentum * self.bias
  
  
  if __name__ == '__main__':
      mndata = MNIST(path=&quot;data&quot;, return_type=&quot;numpy&quot;, mode=&quot;randomly_binarized&quot;)
      X_train, targets_train = mndata.load_training()  # 60.000 images, 28*28 features
      X_val, targets_val = mndata.load_testing()  # 10.000 images, 28*28 features
  
      # Getting the test set splitting the validation set in two equal parts
      # Validation set size decreases from 10.000 to 5000 (of course)
      X_val, X_test = np.split(X_val, 2)  # 5000 images, 28*28 features
      targets_val, targets_test = np.split(targets_val, 2)
  
      targets_train = one_hot(targets_train)
      targets_val = one_hot(targets_val)
      targets_test = one_hot(targets_test)
  
      net = NeuralNetwork()
      d = np.shape(X_train)[1]  # number of features, 28*28
      c = np.shape(targets_train)[0]  # number of classes, 10
  
      # Shallow network with 1 hidden neuron
      # That is 784, 1, 10
      for m in (d, 1, c):
          layer = Layer(m)
          net.add_layer(layer)
  
      net.build()
  
      best_net = net.fit(X_train, targets_train, X_val, targets_val, max_epochs=50)
  
      predictions = best_net.predict(X_test)
      print(&quot;Accuracy score on test set is:&quot;, accuracy_score(targets_test, predictions) * 100, &quot;%&quot;)
  </code></pre>
  <p>Note that:</p>
  <ol>
  <li>The performance are similar even though I set <code>5</code> hidden neurons
  instead of <code>1</code></li>
  <li>Accuracy on training set is very low as well (around <code>11 %</code>)</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-11T09:31:44.070Z
Comments.Text: 5 nodes is basically nothing, have you tried with 500? If the accuracy remain the same then I would start worrying about the code implementation.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-11T10:49:00.297Z
Comments.Text: @EdoardoGuerriero accuracy is low
Comments.UserDisplayName: tail
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-11T15:49:00.183Z
Comments.Text: @EdoardoGuerriero I would disagree with the fact that the number of nodes is not enough. According to the [original mnist paper](http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf) a naiive linear classifier got an *error rate* of roughly 12%. Note that error rate is different than accuracy. Also if accuracy is 11%, then that means it's basically picking a number at random.
Comments.UserDisplayName: juicedatom
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-11T16:10:11.480Z
Comments.Text: Any solution? @juicedatom
Comments.UserDisplayName: tail
Tags: <neural-networks><machine-learning><python><backpropagation><accuracy>
Title: Very low accuracy (0.11) and it remains constant after few epochs on MNIST database
UserDisplayName: tail
ThreadText: # Very low accuracy (0.11) and it remains constant after few epochs on MNIST database
  
  (Asked by: username_0 on 2022-10-09)
  
  I am developing my ANN *from scratch* which is supposed to classify [MNIST database](http://yann.lecun.com/exdb/mnist/) of handwritten digits (0-9). My feed-forward fully connected ANN *has to be* composed of:
  
  1. One input layer, with `28x28 = 784` nodes (that is, features of each image)
  2. One hidden layer, with `1,2,3,4` or `5` nodes (shallow network)
  3. One output layer, with `10` nodes (one for each predicted class)
  
  and *has to* compute gradient w.r.t. weights and bias thanks to backpropagation algorithm and, finally, it should learn exploiting gradient descent with momentum algorithm.
  
  The loss function is: `cross-entropy + softmax`, since the task is about *classification*.
  
  Each hidden neuron is activated by the same activation function, the `sigmoid`; meanwhile the output's neurons are activated by the `identity` function.
  
  The dataset has been divided into:
  
  1. `60.000` training pairs `(image, label)` - for the training
  2. `5000` validation pairs `(image, label)` - for evaluation and select the network which minimize the validation loss
  3. `5000` testing pairs `(image, label)` - for testing the model picked using new metrics such as accuracy
  
  This is the curves I get about training and validation loss: [![training-validation loss](https://i.sstatic.net/tkCf3.png)](https://i.sstatic.net/tkCf3.png) and these are some logs about their loss and accuracy on the validation set:
  
  ```
  E(0) on TrS is: -0.09995663952048875  on VS is: -0.10000215508367913  Accuracy: 9.120000000000001 %
  E(1) on TrS is: -0.09997782058505714  on VS is: -0.10000721545529842  Accuracy: 9.120000000000001 %
  ...
  E(47) on TrS is: -0.1000167196491167  on VS is: -0.10001587893489938  Accuracy: 11.42 %
  E(48) on TrS is: -0.10001676221437443  on VS is: -0.10001594086326895  Accuracy: 11.42 %
  E(49) on TrS is: -0.10001680503178402  on VS is: -0.10001599995199295  Accuracy: 11.42 %
  
  Validation loss is minimum at epoch: 22
  
  ```
  
  Finally, if I test the network at epoch `22`, which is the epoch where the validation loss is minimum, I get the following result:
  
  ```
  Accuracy score on test set is: 11.28 %
  
  ```
  
  These are my questions: as you can see, the accuracy remains (almost) constant and it is *very* low. It has a low value on the test set as well and on the training set. Is that normal? If it is *not*, how can I fix?
  
  This is my code:
  
  ```
  import numpy as np
  from scipy.special import xlog1py
  from scipy.special import expit
  from matplotlib import pyplot as plt
  from mnist.loader import MNIST
  
  def softmax(y):
      e_y = np.exp(y - np.max(y))
      return e_y / e_y.sum()
  
  def cross_entropy(y, t, derivative=False, post_process=True):
      if post_process:
          if derivative:
              return y - t
          return -np.sum(np.sum(xlog1py(t, softmax(y)), axis=0))
  
  def sigmoid(a, derivative=False):
      f_a = expit(-a)
      if derivative:
          return np.multiply(f_a, (1 - f_a))
      return f_a
  
  def identity(a, derivative=False):
      f_a = a
      if derivative:
          return np.ones(np.shape(a))
      return f_a
  
  def accuracy_score(targets, predictions):
      correct_predictions = 0
      for item in range(np.shape(predictions)[1]):
          # print(predictions[:, item])  # dbg
          argmax_idx = np.argmax(predictions[:, item])
          # print("argmax idx", argmax_idx)  # dbg
          # print(targets_test[:, item])  # dbg
          if targets[argmax_idx, item] == 1:
              correct_predictions += 1
      return correct_predictions / np.shape(predictions)[1]
  
  def one_hot(targets):
      return np.asmatrix(np.eye(np.max(targets) + 1)[targets]).T
  
  def plot(epochs, loss_train, loss_val):
      plt.plot(epochs, loss_train)
      plt.plot(epochs, loss_val, color="orange")
      plt.legend(["Training Loss", "Validation Loss"])
      plt.xlabel("Epochs")
      plt.ylabel("Loss")
      plt.grid(True)
      plt.show()
  
  class NeuralNetwork:
  
      def __init__(self):
          self.layers = []
  
      def add_layer(self, layer):
          self.layers.append(layer)
  
      def build(self):
          for i, layer in enumerate(self.layers):
              if i == 0:
                  layer.type = "input"
              else:
                  layer.type = "output" if i == len(self.layers) - 1 else "hidden"
                  layer.configure(self.layers[i - 1].neurons)
  
      def fit(self, X_train, targets_train, X_val, targets_val, max_epochs=50):
          e_loss_train = []
          e_loss_val = []
  
          # Getting the minimum loss on validation set
          predictions_val = self.predict(X_val)
          min_loss_val = cross_entropy(predictions_val, targets_val)
  
          best_net = self  # net which minimize validation loss
          best_epoch = 0  # epoch where the validation loss is minimum
  
          # batch mode
          for epoch in range(max_epochs):
              predictions_train = self.predict(X_train)
              self.back_prop(targets_train, cross_entropy)
              self.learning_rule(l_rate=0.000001, momentum=0.9)
              loss_train = cross_entropy(predictions_train, targets_train)
              e_loss_train.append(loss_train)
  
              # Validation
              predictions_val = self.predict(X_val)
              loss_val = cross_entropy(predictions_val, targets_val)
              e_loss_val.append(loss_val)
  
              print("E(%d) on TrS is:" % epoch, loss_train, " on VS is:", loss_val, " Accuracy:",
                    accuracy_score(targets_val, predictions_val) * 100, "%")
  
              if loss_val < min_loss_val:
                  min_loss_val = loss_val
                  best_epoch = epoch
                  best_net = self
  
          print("Validation loss is minimum at epoch:", best_epoch)
  
          plot(np.arange(max_epochs), e_loss_train, e_loss_val)
  
          return best_net
  
      # Matrix of predictions where the i-th column corresponds to the i-th item
      def predict(self, dataset):
          z = dataset.T
          for layer in self.layers:
              z = layer.forward_prop_step(z)
          return z
  
      def back_prop(self, target, loss):
          for i, layer in enumerate(self.layers[:0:-1]):
              next_layer = self.layers[-i]
              prev_layer = self.layers[-i - 2]
              layer.back_prop_step(next_layer, prev_layer, target, loss)
  
      def learning_rule(self, l_rate, momentum):
          # Momentum GD
          for layer in [layer for layer in self.layers if layer.type != "input"]:
              layer.update_weights(l_rate, momentum)
              layer.update_bias(l_rate, momentum)
  
  class Layer:
  
      def __init__(self, neurons, type=None, activation=None):
          self.dE_dW = None  # derivatives dE/dW where W is the weights matrix
          self.dE_db = None  # derivatives dE/db where b is the bias
          self.dact_a = None  # derivative of the activation function
          self.out = None  # layer output
          self.weights = None  # input weights
          self.bias = None  # layer bias
          self.w_sum = None  # weighted_sum
          self.neurons = neurons  # number of neurons
          self.type = type  # input, hidden or output
          self.activation = activation  # activation function
          self.deltas = None  # for back-prop
  
      def configure(self, prev_layer_neurons):
          self.set_activation()
          self.weights = np.asmatrix(np.random.normal(-0.1, 0.02, (self.neurons, prev_layer_neurons)))
          self.bias = np.asmatrix(np.random.normal(-0.1, 0.02, self.neurons)).T  # vettore colonna
  
      def set_activation(self):
          if self.activation is None:
              if self.type == "hidden":
                  self.activation = sigmoid
              elif self.type == "output":
                  self.activation = identity
  
      def forward_prop_step(self, z):
          if self.type == "input":
              self.out = z
          else:
              self.w_sum = np.dot(self.weights, z) + self.bias
              self.out = self.activation(self.w_sum)
          return self.out
  
      def back_prop_step(self, next_layer, prev_layer, target, local_loss):
          if self.type == "output":
              self.dact_a = self.activation(self.w_sum, derivative=True)
              self.deltas = np.multiply(self.dact_a,
                                        local_loss(self.out, target, derivative=True))
          else:
              self.dact_a = self.activation(self.w_sum, derivative=True)  # (m,batch_size)
              self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weights.T, next_layer.deltas))
  
          self.dE_dW = self.deltas * prev_layer.out.T
  
          self.dE_db = np.sum(self.deltas, axis=1)
  
      def update_weights(self, l_rate, momentum):
          # Momentum GD
          self.weights = self.weights - l_rate * self.dE_dW
          self.weights = -l_rate * self.dE_dW + momentum * self.weights
  
      def update_bias(self, l_rate, momentum):
          # Momentum GD
          self.bias = self.bias - l_rate * self.dE_db
          self.bias = -l_rate * self.dE_db + momentum * self.bias
  
  if __name__ == '__main__':
      mndata = MNIST(path="data", return_type="numpy", mode="randomly_binarized")
      X_train, targets_train = mndata.load_training()  # 60.000 images, 28*28 features
      X_val, targets_val = mndata.load_testing()  # 10.000 images, 28*28 features
  
      # Getting the test set splitting the validation set in two equal parts
      # Validation set size decreases from 10.000 to 5000 (of course)
      X_val, X_test = np.split(X_val, 2)  # 5000 images, 28*28 features
      targets_val, targets_test = np.split(targets_val, 2)
  
      targets_train = one_hot(targets_train)
      targets_val = one_hot(targets_val)
      targets_test = one_hot(targets_test)
  
      net = NeuralNetwork()
      d = np.shape(X_train)[1]  # number of features, 28*28
      c = np.shape(targets_train)[0]  # number of classes, 10
  
      # Shallow network with 1 hidden neuron
      # That is 784, 1, 10
      for m in (d, 1, c):
          layer = Layer(m)
          net.add_layer(layer)
  
      net.build()
  
      best_net = net.fit(X_train, targets_train, X_val, targets_val, max_epochs=50)
  
      predictions = best_net.predict(X_test)
      print("Accuracy score on test set is:", accuracy_score(targets_test, predictions) * 100, "%")
  
  ```
  
  Note that:
  
  1. The performance are similar even though I set `5` hidden neurons
     instead of `1`
  2. Accuracy on training set is very low as well (around `11 %`)
  
  **Tags:** neural-networks, machine-learning, python, backpropagation, accuracy
  
  > **Comments:**
  > **username_1**: 5 nodes is basically nothing, have you tried with 500? If the accuracy remain the same then I would start worrying about the code implementation.  
  > **username_0**: @username_1 accuracy is low  
  > **username_2**: @username_1 I would disagree with the fact that the number of nodes is not enough. According to the [original mnist paper](http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf) a naiive linear classifier got an *error rate* of roughly 12%. Note that error rate is different than accuracy. Also if accuracy is 11%, then that means it's basically picking a number at random.  
  > **username_0**: Any solution? @username_2  
  
  ---
  
  ## 0 answers
  
  