Answers.Body: <p>It seems like you're suffering from the the dying ReLU problem. ReLU enforces positive values so the weights and biases your network learned are leading to a negative value passed through the ReLU function - meaning you would get 0. There are a few things you can do. I do not know the exact format of your data, but if it is MNIST it is possible you simply don't have normalized values. You could be learning a large negative bias as a result. Try dividing every pixel intensity in your dataset by the float 255.0 to normalize your values and see if that fixes your problem.</p>
  
  <p>You could also change your activation function to something such as <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Leaky_ReLU" rel="nofollow noreferrer">Leaky ReLU</a> which attempts to solve this problem with a small positive gradient for negative values. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-17T11:23:51.900Z
Answers.Text: I have already normalised the data between 0 and 1,however i will try out the leaky ReLU.
Answers.UserDisplayName: finlay morrison
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T10:36:08.557Z
Answers.LastActivityDate: 2020-04-16T10:36:08.557Z
Answers.UserDisplayName: Archie Shahidullah
Body: <p>I'm not really sure if this is the sort of question to ask on here, since it is less of a general question about AI and more about the coding of it, however I thought it wouldn't fit on stack overflow.</p>
  
  <p>I have been programming a multilayer perceptron in c++, and it seems to be working with a sigmoid function, however when I change the activation function to ReLU it does not converge and stays at an average cost of 1 per training example. this is because all of the network's output neurons output a 0.</p>
  
  <p>With the sigmoid function it converges rather nicely, I did a bit of testing and after about 1000 generations it got to an average cost of 0.1 on the first 1000 items in the MNIST dataset.</p>
  
  <p>I will show you the code I changes first for the activation functions, and then i will put the whole block of code in.</p>
  
  <p>Any help would be greatly appreciated!</p>
  
  <p>Sigmoid:</p>
  
  
  
  <pre><code>inline float activation(float num)
  {
      return 1 / (1 + std::exp(-num));
  }
  
  inline float activation_derivative(float num)
  {
      return activation(num) * (1 - activation(num));
  }
  </code></pre>
  
  <p>ReLU:</p>
  
  
  
  <pre><code>inline float activation(float num)
  {
      return std::max(num, 0.0f);
  }
  
  inline float activation_derivative(float num)
  {
      return num &gt; 0 ? 1.0f : 0.0f;
  }
  </code></pre>
  
  <p>And here's the whole block of code (I collapsed the region of code for benchmarking and the region for creating the dataset):</p>
  
  
  
  <pre><code>#include &lt;iostream&gt;
  #include &lt;fstream&gt;
  #include &lt;vector&gt;
  #include &lt;random&gt;
  #include &lt;chrono&gt;
  #include &lt;cmath&gt;
  #include &lt;string&gt;
  #include &lt;algorithm&gt;
  
  #pragma region benchmarking
  #pragma endregion
  
  class Network
  {
  public:
      float cost = 0.0f;
      std::vector&lt;std::vector&lt;std::vector&lt;float&gt;&gt;&gt; weights;
      std::vector&lt;std::vector&lt;std::vector&lt;float&gt;&gt;&gt; deriv_weights;
      std::vector&lt;std::vector&lt;float&gt;&gt; biases;
      std::vector&lt;std::vector&lt;float&gt;&gt; deriv_biases;
      std::vector&lt;std::vector&lt;float&gt;&gt; activations;
      std::vector&lt;std::vector&lt;float&gt;&gt; deriv_activations;
      void clear_deriv_activations()
      {
          for (unsigned int i = 0; i &lt; deriv_activations.size(); ++i)
          {
              std::fill(deriv_activations[i].begin(), deriv_activations[i].end(), 0.0f);
          }
      }
      int get_memory_usage()
      {
          int memory = 4;
          memory += get_vector_memory_usage(weights);
          memory += get_vector_memory_usage(deriv_weights);
          memory += get_vector_memory_usage(biases);
          memory += get_vector_memory_usage(deriv_biases);
          memory += get_vector_memory_usage(activations);
          memory += get_vector_memory_usage(deriv_activations);
          return memory;
      }
  };
  
  struct DataSet
  {
      std::vector&lt;std::vector&lt;float&gt;&gt; training_inputs;
      std::vector&lt;std::vector&lt;float&gt;&gt; training_answers;
      std::vector&lt;std::vector&lt;float&gt;&gt; testing_inputs;
      std::vector&lt;std::vector&lt;float&gt;&gt; testing_answers;
  };
  
  
  Network create_network(std::vector&lt;int&gt; layers)
  {
      Network network;
      int layer_count = layers.size() - 1;
      network.weights.reserve(layer_count);
      network.deriv_weights.reserve(layer_count);
      network.biases.reserve(layer_count);
      network.deriv_biases.reserve(layer_count);
      network.activations.reserve(layer_count);
      network.deriv_activations.reserve(layer_count);
      int nodes_in_prev_layer = layers[0];
      for (unsigned int i = 0; i &lt; layers.size() - 1; ++i)
      {
          int nodes_in_layer = layers[i + 1];
          network.weights.emplace_back();
          network.weights[i].reserve(nodes_in_layer);
          network.deriv_weights.emplace_back();
          network.deriv_weights[i].reserve(nodes_in_layer);
          network.biases.emplace_back();
          network.biases[i].reserve(nodes_in_layer);
          network.deriv_biases.emplace_back(nodes_in_layer, 0.0f);
          network.activations.emplace_back(nodes_in_layer, 0.0f);
          network.deriv_activations.emplace_back(nodes_in_layer, 0.0f);
          for (int j = 0; j &lt; nodes_in_layer; ++j)
          {
              network.weights[i].emplace_back();
              network.weights[i][j].reserve(nodes_in_prev_layer);
              network.deriv_weights[i].emplace_back(nodes_in_prev_layer, 0.0f);
              for (int k = 0; k &lt; nodes_in_prev_layer; ++k)
              {
                  float input_weight = (2 * (float(std::rand()) / RAND_MAX)) - 1; 
                  network.weights[i][j].push_back(input_weight);
              }
              float input_bias = (2 * (float(std::rand()) / RAND_MAX)) - 1;
              network.biases[i].push_back(input_bias);
          }
          nodes_in_prev_layer = nodes_in_layer;
      }
      return network;
  }
  
  void judge_network(Network &amp;network, const std::vector&lt;float&gt;&amp; correct_answers)
  {
      int final_layer_index = network.activations.size() - 1;
      for (unsigned int i = 0; i &lt; network.activations[final_layer_index].size(); ++i)
      {
          float val_sq = (network.activations[final_layer_index][i] - correct_answers[i]);
          network.cost += val_sq * val_sq;
      }
  }
  
  inline float activation(float num)
  {
      return std::max(num, 0.0f);
  }
  
  void forward_propogate(Network&amp; network, const std::vector&lt;float&gt;&amp; input)
  {
      const std::vector&lt;float&gt;* last_layer_activations = &amp;input;
      int last_layer_node_count = input.size();
      for (unsigned int i = 0; i &lt; network.weights.size(); ++i)
      {
          for (unsigned int j = 0; j &lt; network.weights[i].size(); ++j)
          {
              float total = network.biases[i][j];
              for (int k = 0; k &lt; last_layer_node_count; ++k)
              {
                  total +=  (*last_layer_activations)[k] * network.weights[i][j][k];
              }
              network.activations[i][j] = activation(total);
          }
          last_layer_activations = &amp;network.activations[i];
          last_layer_node_count = network.weights[i].size();
      }
  }
  
  void final_layer_deriv_activations(Network&amp; network, const std::vector&lt;float&gt;&amp; correct_answers)
  {
      int final_layer_index = network.activations.size() - 1;
      int final_layer_node_count = network.activations[final_layer_index].size();
      for (int i = 0; i &lt; final_layer_node_count; ++i)
      {
          float deriv = network.activations[final_layer_index][i] - correct_answers[i];
          network.deriv_activations[final_layer_index][i] = deriv * 2;
      }
  }
  
  inline float activation_derivative(float num)
  {
      return num &gt; 0 ? 1.0f : 0.0f;
  }
  
  void back_propogate_layer(Network&amp; network, int layer)
  {
      int nodes_in_layer = network.activations[layer].size();
      int nodes_in_prev_layer = network.activations[layer - 1].size();
      for (int i = 0; i &lt; nodes_in_layer; ++i)
      {
          float total = network.biases[layer][i];
          for (int j = 0; j &lt; nodes_in_prev_layer; ++j)
          {
              total += network.weights[layer][i][j] * network.activations[layer - 1][j];
          }
          float dzda = activation_derivative(total);
          float dzdc = dzda * network.deriv_activations[layer][i];
          for (int j = 0; j &lt; nodes_in_prev_layer; ++j)
          {
              network.deriv_weights[layer][i][j] += network.activations[layer - 1][j] * dzdc;
              network.deriv_activations[layer - 1][j] += network.weights[layer][i][j] * dzdc;
          }
          network.deriv_biases[layer][i] += dzdc;
      }
  }
  
  void back_propogate_first_layer(Network&amp; network, std::vector&lt;float&gt; inputs)
  {
      int nodes_in_layer = network.activations[0].size();
      int input_count = inputs.size();
      for (int i = 0; i &lt; nodes_in_layer; ++i)
      {
          float total = network.biases[0][i];
          for (int j = 0; j &lt; input_count; ++j)
          {
              total += network.weights[0][i][j] * inputs[j];
          }
          float dzda = activation_derivative(total);
          float dzdc = dzda * network.deriv_activations[0][i];
          for (int j = 0; j &lt; input_count; ++j)
          {
              network.deriv_weights[0][i][j] += inputs[j] * dzdc;
          }
          network.deriv_biases[0][i] += dzdc;
      }
  }
  
  void back_propogate(Network&amp; network, const std::vector&lt;float&gt;&amp; inputs, const std::vector&lt;float&gt;&amp; correct_answers)
  {
      network.clear_deriv_activations();
      final_layer_deriv_activations(network, correct_answers);
      for (int i = network.activations.size() - 1; i &gt; 0; --i)
      {
          back_propogate_layer(network, i);
      }
      back_propogate_first_layer(network, inputs);
  }
  
  void apply_derivatives(Network&amp; network, int training_example_count)
  {
      for (unsigned int i = 0; i &lt; network.weights.size(); ++i)
      {
          for (unsigned int j = 0; j &lt; network.weights[i].size(); ++j)
          {
              for (unsigned int k = 0; k &lt; network.weights[i][j].size(); ++k)
              {
                  network.weights[i][j][k] -= network.deriv_weights[i][j][k] / training_example_count;
                  network.deriv_weights[i][j][k] = 0;
              }
              network.biases[i][j] -= network.deriv_biases[i][j] / training_example_count;
              network.deriv_biases[i][j] = 0;
              network.deriv_activations[i][j] = 0;
          }
      }
  }
  
  void training_iteration(Network&amp; network, const DataSet&amp; data)
  {
      int training_example_count = data.training_inputs.size();
      for (int i = 0; i &lt; training_example_count; ++i)
      {
          forward_propogate(network, data.training_inputs[i]);
          judge_network(network, data.training_answers[i]);
          back_propogate(network, data.training_inputs[i], data.training_answers[i]);
      }
      apply_derivatives(network, training_example_count);
  }
  
  void train_network(Network&amp; network, const DataSet&amp; dataset, int training_iterations)
  {
      for (int i = 0; i &lt; training_iterations; ++i)
      {
          training_iteration(network, dataset);
          std::cout &lt;&lt; "Generation " &lt;&lt; i &lt;&lt; ": " &lt;&lt; network.cost &lt;&lt; std::endl;
          network.cost = 0.0f;
      }
  }
  
  #pragma region dataset creation
  
  #pragma endregion
  
  int main() 
  {
      Timer timer;
      DataSet dataset = create_dataset_from_file("data.txt");
      Network network = create_network({784, 128, 10});
      train_network(network, dataset, 1000);
      std::cout &lt;&lt; timer.get_duration() &lt;&lt; std::endl;
      std::cin.get();
  }
  <span class="math-container">```</span>
  </code></pre>
  
Tags: <convergence><relu><c++><sigmoid>
Title: Neural network doesn't seem to converge with ReLU but it does with Sigmoid?
UserDisplayName: finlay morrison
ThreadText: # Neural network doesn't seem to converge with ReLU but it does with Sigmoid?
  
  (Asked by: username_0 on 2020-04-15)
  
  I'm not really sure if this is the sort of question to ask on here, since it is less of a general question about AI and more about the coding of it, however I thought it wouldn't fit on stack overflow.
  I have been programming a multilayer perceptron in c++, and it seems to be working with a sigmoid function, however when I change the activation function to ReLU it does not converge and stays at an average cost of 1 per training example. this is because all of the network's output neurons output a 0.
  With the sigmoid function it converges rather nicely, I did a bit of testing and after about 1000 generations it got to an average cost of 0.1 on the first 1000 items in the MNIST dataset.
  I will show you the code I changes first for the activation functions, and then i will put the whole block of code in.
  Any help would be greatly appreciated!
  Sigmoid:
  inline float activation(float num)
  {
      return 1 / (1 + std::exp(-num));
  }
  
  inline float activation_derivative(float num)
  {
      return activation(num) * (1 - activation(num));
  }
  
  ReLU:
  inline float activation(float num)
  {
      return std::max(num, 0.0f);
  }
  
  inline float activation_derivative(float num)
  {
      return num > 0 ? 1.0f : 0.0f;
  }
  
  And here's the whole block of code (I collapsed the region of code for benchmarking and the region for creating the dataset):
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  
  #pragma region benchmarking
  #pragma endregion
  
  class Network
  {
  public:
      float cost = 0.0f;
      std::vector>> weights;
      std::vector>> deriv_weights;
      std::vector> biases;
      std::vector> deriv_biases;
      std::vector> activations;
      std::vector> deriv_activations;
      void clear_deriv_activations()
      {
          for (unsigned int i = 0; i < deriv_activations.size(); ++i)
          {
              std::fill(deriv_activations[i].begin(), deriv_activations[i].end(), 0.0f);
          }
      }
      int get_memory_usage()
      {
          int memory = 4;
          memory += get_vector_memory_usage(weights);
          memory += get_vector_memory_usage(deriv_weights);
          memory += get_vector_memory_usage(biases);
          memory += get_vector_memory_usage(deriv_biases);
          memory += get_vector_memory_usage(activations);
          memory += get_vector_memory_usage(deriv_activations);
          return memory;
      }
  };
  
  struct DataSet
  {
      std::vector> training_inputs;
      std::vector> training_answers;
      std::vector> testing_inputs;
      std::vector> testing_answers;
  };
  
  
  Network create_network(std::vector layers)
  {
      Network network;
      int layer_count = layers.size() - 1;
      network.weights.reserve(layer_count);
      network.deriv_weights.reserve(layer_count);
      network.biases.reserve(layer_count);
      network.deriv_biases.reserve(layer_count);
      network.activations.reserve(layer_count);
      network.deriv_activations.reserve(layer_count);
      int nodes_in_prev_layer = layers[0];
      for (unsigned int i = 0; i < layers.size() - 1; ++i)
      {
          int nodes_in_layer = layers[i + 1];
          network.weights.emplace_back();
          network.weights[i].reserve(nodes_in_layer);
          network.deriv_weights.emplace_back();
          network.deriv_weights[i].reserve(nodes_in_layer);
          network.biases.emplace_back();
          network.biases[i].reserve(nodes_in_layer);
          network.deriv_biases.emplace_back(nodes_in_layer, 0.0f);
          network.activations.emplace_back(nodes_in_layer, 0.0f);
          network.deriv_activations.emplace_back(nodes_in_layer, 0.0f);
          for (int j = 0; j < nodes_in_layer; ++j)
          {
              network.weights[i].emplace_back();
              network.weights[i][j].reserve(nodes_in_prev_layer);
              network.deriv_weights[i].emplace_back(nodes_in_prev_layer, 0.0f);
              for (int k = 0; k < nodes_in_prev_layer; ++k)
              {
                  float input_weight = (2 * (float(std::rand()) / RAND_MAX)) - 1; 
                  network.weights[i][j].push_back(input_weight);
              }
              float input_bias = (2 * (float(std::rand()) / RAND_MAX)) - 1;
              network.biases[i].push_back(input_bias);
          }
          nodes_in_prev_layer = nodes_in_layer;
      }
      return network;
  }
  
  void judge_network(Network &network, const std::vector& correct_answers)
  {
      int final_layer_index = network.activations.size() - 1;
      for (unsigned int i = 0; i < network.activations[final_layer_index].size(); ++i)
      {
          float val_sq = (network.activations[final_layer_index][i] - correct_answers[i]);
          network.cost += val_sq * val_sq;
      }
  }
  
  inline float activation(float num)
  {
      return std::max(num, 0.0f);
  }
  
  void forward_propogate(Network& network, const std::vector& input)
  {
      const std::vector* last_layer_activations = &input
      int last_layer_node_count = input.size();
      for (unsigned int i = 0; i < network.weights.size(); ++i)
      {
          for (unsigned int j = 0; j < network.weights[i].size(); ++j)
          {
              float total = network.biases[i][j];
              for (int k = 0; k < last_layer_node_count; ++k)
              {
                  total +=  (*last_layer_activations)[k] * network.weights[i][j][k];
              }
              network.activations[i][j] = activation(total);
          }
          last_layer_activations = &network.activations[i];
          last_layer_node_count = network.weights[i].size();
      }
  }
  
  void final_layer_deriv_activations(Network& network, const std::vector& correct_answers)
  {
      int final_layer_index = network.activations.size() - 1;
      int final_layer_node_count = network.activations[final_layer_index].size();
      for (int i = 0; i < final_layer_node_count; ++i)
      {
          float deriv = network.activations[final_layer_index][i] - correct_answers[i];
          network.deriv_activations[final_layer_index][i] = deriv * 2;
      }
  }
  
  inline float activation_derivative(float num)
  {
      return num > 0 ? 1.0f : 0.0f;
  }
  
  void back_propogate_layer(Network& network, int layer)
  {
      int nodes_in_layer = network.activations[layer].size();
      int nodes_in_prev_layer = network.activations[layer - 1].size();
      for (int i = 0; i < nodes_in_layer; ++i)
      {
          float total = network.biases[layer][i];
          for (int j = 0; j < nodes_in_prev_layer; ++j)
          {
              total += network.weights[layer][i][j] * network.activations[layer - 1][j];
          }
          float dzda = activation_derivative(total);
          float dzdc = dzda * network.deriv_activations[layer][i];
          for (int j = 0; j < nodes_in_prev_layer; ++j)
          {
              network.deriv_weights[layer][i][j] += network.activations[layer - 1][j] * dzdc;
              network.deriv_activations[layer - 1][j] += network.weights[layer][i][j] * dzdc;
          }
          network.deriv_biases[layer][i] += dzdc;
      }
  }
  
  void back_propogate_first_layer(Network& network, std::vector inputs)
  {
      int nodes_in_layer = network.activations[0].size();
      int input_count = inputs.size();
      for (int i = 0; i < nodes_in_layer; ++i)
      {
          float total = network.biases[0][i];
          for (int j = 0; j < input_count; ++j)
          {
              total += network.weights[0][i][j] * inputs[j];
          }
          float dzda = activation_derivative(total);
          float dzdc = dzda * network.deriv_activations[0][i];
          for (int j = 0; j < input_count; ++j)
          {
              network.deriv_weights[0][i][j] += inputs[j] * dzdc;
          }
          network.deriv_biases[0][i] += dzdc;
      }
  }
  
  void back_propogate(Network& network, const std::vector& inputs, const std::vector& correct_answers)
  {
      network.clear_deriv_activations();
      final_layer_deriv_activations(network, correct_answers);
      for (int i = network.activations.size() - 1; i > 0; --i)
      {
          back_propogate_layer(network, i);
      }
      back_propogate_first_layer(network, inputs);
  }
  
  void apply_derivatives(Network& network, int training_example_count)
  {
      for (unsigned int i = 0; i < network.weights.size(); ++i)
      {
          for (unsigned int j = 0; j < network.weights[i].size(); ++j)
          {
              for (unsigned int k = 0; k < network.weights[i][j].size(); ++k)
              {
                  network.weights[i][j][k] -= network.deriv_weights[i][j][k] / training_example_count;
                  network.deriv_weights[i][j][k] = 0;
              }
              network.biases[i][j] -= network.deriv_biases[i][j] / training_example_count;
              network.deriv_biases[i][j] = 0;
              network.deriv_activations[i][j] = 0;
          }
      }
  }
  
  void training_iteration(Network& network, const DataSet& data)
  {
      int training_example_count = data.training_inputs.size();
      for (int i = 0; i < training_example_count; ++i)
      {
          forward_propogate(network, data.training_inputs[i]);
          judge_network(network, data.training_answers[i]);
          back_propogate(network, data.training_inputs[i], data.training_answers[i]);
      }
      apply_derivatives(network, training_example_count);
  }
  
  void train_network(Network& network, const DataSet& dataset, int training_iterations)
  {
      for (int i = 0; i < training_iterations; ++i)
      {
          training_iteration(network, dataset);
          std::cout << "Generation " << i << ": " << network.cost << std::endl;
          network.cost = 0.0f;
      }
  }
  
  #pragma region dataset creation
  
  #pragma endregion
  
  int main() 
  {
      Timer timer;
      DataSet dataset = create_dataset_from_file("data.txt");
      Network network = create_network({784, 128, 10});
      train_network(network, dataset, 1000);
      std::cout << timer.get_duration() << std::endl;
      std::cin.get();
  }
  ```
  
  **Tags:** convergence, relu, c++, sigmoid
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-16. Score: 1
  
  It seems like you're suffering from the the dying ReLU problem. ReLU enforces positive values so the weights and biases your network learned are leading to a negative value passed through the ReLU function - meaning you would get 0. There are a few things you can do. I do not know the exact format of your data, but if it is MNIST it is possible you simply don't have normalized values. You could be learning a large negative bias as a result. Try dividing every pixel intensity in your dataset by the float 255.0 to normalize your values and see if that fixes your problem.
  
  You could also change your activation function to something such as [Leaky ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Leaky_ReLU) which attempts to solve this problem with a small positive gradient for negative values.
  
  > **Comments:**
  > **username_0**: I have already normalised the data between 0 and 1,however i will try out the leaky ReLU.  
  
  ---
  
  