Answers.Body: <p>Unfortunately I highly doubt anyone is going to check your code or exact implementation as that is a very time consuming process. Luckily for you though, you don't need someone to check, you can check it yourself with <strong>gradient checking</strong>! The basic process is to manually calculate the exact gradient for every single parameter for an input by varying every parameter one at a time, then seeing how each changes your output. With a proper implementation of backpropagation, the gradients will match.</p>
  <p><a href="https://stackoverflow.com/a/47506918/9546874">Here's another question describing this process on stack overflow</a></p>
  <p>Note though that with ReLU the function is undefined at 0, and you will quickly see how this is a problem when you start doing grad checking. The gradients wont match if you try and check a parameter using the output of a ReLU function when the input to said ReLU is 0. I did some research on this a while ago when I had this problem, and I believe the consensus was to just ignore it</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-06T02:02:37.913Z
Answers.Text: You're right. I don't know why I thought anyone would. I will try this method, thank you.
Answers.UserDisplayName: denvr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-06T01:21:28.707Z
Answers.LastActivityDate: 2022-10-06T01:21:28.707Z
Answers.UserDisplayName: Recessive
Body: <p>I recently just finished programming a neural network in c#, and it seems like it's working. My question is if I'm doing it right. It's a very confusing process so I will explain.</p>
  <p>Basically every neuron in the network has a bias (except the first layer) and fully connected weights to the next. every neuron also has a <code>double[] desiredBias</code> which holds all the desired changes to the neuron for every iteration in a given mini batch. The same thing for the weights. So I start from the last layer, and from there I calculate the &quot;sensitivity&quot; you might say between the cost of a given neuron on that last layer and the bias, and i take note of it (put it into changes). Then for every weight connected to the layer behind, I also calculate the sensitivity for that one. And I keep doing that until I get to the second to first layer. And to take note of the <code>desiredChanges</code> for neurons not in the last layer, I use this array called &quot;<code>changes</code>&quot;. For the first layer it's basically just a list of all the derivatives of the activations (z) with respect to the actual activations (sigmoid(z)) respectively. After I shift to the layer behind, and behind, every time I do that, I create a new changes array, which is the size of the layer behind, and for each item in the previous changes array, I take the sum of all of them multiplied by the partial derivative of that same z-activation with respect to the previous layer's z activation. I do this for every neuron in the previous layer.</p>
  <p>after all of that I end up with for every neuron a list of desired changes to it's bias, and a list of lists which contain desired changes to the weights of each neuron. Then for each neuron (except for the first of course) I average together all of the biases and SUBTRACT it from the neuron's bias. I do the same thing for the weights.</p>
  <p>Is this the correct process to go through for exactly one batch of training data?</p>
  <p>here's my code:</p>
  <pre><code>using System;
  
  namespace NeuralNetwork {
      
      public class Functions {
          
          // Activation Functions
          
          public static double sigmoid(double x) {
              return 1 / (1 + Math.Pow(Math.E, -x));
          }
          public static double ReLU(double x) {
              return Math.Max(0, x);
          }
          public static double Tanh(double x) {
              return Math.Tanh(x);
          }
          public static double Linear(double x) {
              return x;
          }
          
          // Derivatives
          
          public static double sigmoidDerivative(double x) {
              return sigmoid(x) * (1 - sigmoid(x));
          }
          public static double ReLUDerivative(double x) {
              return (x &gt; 0) ? 1 : 0;
          }
          public static double TanhDerivative(double x) {
              return 1 - (Math.Pow(Tanh(x) * Tanh(x), 2));
          }
          public static double LinearDerivative(double x) {
              return 1;
          }
          
          public enum ActFunction {
              Sigmoid,
              ReLU,
              Tanh,
              Linear
          }
          
          // Loss Functions
          
          public static double MSE(double[] outputs, double[] desired) {
              double sum = 0;
              for (int i = 0; i &lt; outputs.Length; i++) {
                  sum += Math.Pow((outputs[i] - desired[i]), 2);
              }
              return sum / outputs.Length;
          }
          
          // Derivatives
          
          public static double[] MSEDerivative(double[] outputs, double[] desired) {
              double[] derivatives = new double[outputs.Length];
              for (int i = 0; i &lt; outputs.Length; i++) {
                  derivatives[i] = 2 * (outputs[i] - desired[i]);
              }
              return derivatives;
          }
          
          public enum LossFunction {
              MSE
          }
          
      }
      
      public class Neuron {
      
          public double activation = 0;
          public Functions.ActFunction activationFunction = Functions.ActFunction.Linear;
          
          public double bias;
          public double[] weights = new double[] {};
          public int x;
          public int y;
          public Neuron[][] network;
          
          public double[] desiredBias;
          public double[][] desiredWeights;
          
          public void fire() {
              for (int i = 0; i &lt; network[x + 1].Length; i++) {
                  Neuron next = network[x + 1][i];
                  next.activation += activation * weights[i];
              }
          }
      
          public void activate() {
              double n_activation = activation + bias;
              
              switch(activationFunction) {
                  case Functions.ActFunction.Sigmoid:
                      activation = Functions.sigmoid(n_activation);
                      break;
                  case Functions.ActFunction.ReLU:
                      activation = Functions.ReLU(n_activation);
                      break;
                  case Functions.ActFunction.Tanh:
                      activation = Functions.Tanh(n_activation);
                      break;
                  case Functions.ActFunction.Linear:
                      activation = Functions.Linear(n_activation);
                      break;
              }
          }
          
      }
      
      public class Network {
      
          public Neuron[][] network;
          public double learningRate;
          public int iterations;
          public Functions.LossFunction lossFunction;
          public int trainingLength;
      
          public Network(int[] size, double _learningRate, int _iterations, int _trainingLength, Functions.LossFunction _lossFunction) {
              learningRate = _learningRate;
              iterations = _iterations;
              trainingLength = _trainingLength;
              lossFunction = _lossFunction;
              network = new Neuron[size.Length][];
              for (int i = 0; i &lt; size.Length; i++) {
                  network[i] = new Neuron[size[i]];
              }
              for (int i = 0; i &lt; size.Length; i++) { 
                  for (int j = 0; j &lt; size[i]; j++) { 
                      network[i][j] = new Neuron();
                  }
              }
              for (int i = 0; i &lt; network.Length - 1; i++) {
                  for (int j = 0; j &lt; network[i].Length; j++) { 
                      Neuron neuron = network[i][j];
                      neuron.weights = new double[network[i + 1].Length];
                      neuron.desiredWeights = new double[neuron.weights.Length][]; // i would have put iterations right here but the compiler thinks otherwise
                      for (int k = 0; k &lt; neuron.desiredWeights.Length; k++) {
                          neuron.desiredWeights[k] = new double[trainingLength/iterations];
                      }
                  }
              }
              for (int i = 1; i &lt; network.Length; i++) {
                  for (int j = 0; j &lt; network[i].Length; j++) { 
                      Neuron neuron = network[i][j];
                      neuron.desiredBias = new double[trainingLength/iterations];
                  }
              }
          }
      
          public void init() {
              Random rand = new Random();
              for (int i = 0; i &lt; network.Length; i++) {
                  for (int j = 0; j &lt; network[i].Length; j++) {
                      Neuron neuron = network[i][j];
                      neuron.x = i;
                      neuron.y = j;
                      neuron.network = network;
                      
                      if (i != 0) {
                          neuron.bias = (rand.NextDouble() * 2) - 1;
                      }
                      
                      if (i != network.Length - 1) {
                          for (int k = 0; k &lt; network[neuron.x + 1].Length; k++) {
                              neuron.weights[k] = (rand.NextDouble() * 20) - 1;
                          }
                      }
                  }
              }
          }
          
          public void info() {
              Console.WriteLine(&quot;Here are the statistics for the network:\n&quot;);
              
              for (int i = 0; i &lt; network.Length; i++) {
                  for (int j = 0; j &lt; network[i].Length; j++) {
                      Neuron neuron = network[i][j];
                      Console.WriteLine(&quot;Neuron &quot; + j + &quot; of layer &quot; + i);
                      //Console.WriteLine(&quot;Activation: &quot; + neuron.activation);
                      Console.WriteLine(&quot;Bias: &quot; + neuron.bias);
                      Console.WriteLine(&quot;X and Y: &quot; + neuron.x + &quot;, &quot; + neuron.y);
                      string weights = &quot;Weights: &quot;;
                      foreach (double weight in neuron.weights) {
                          weights += weight + &quot; &quot;;
                      }
                      Console.WriteLine(weights);
                      Console.WriteLine();
                  }
              }
              
              Console.WriteLine();
          }
          
          public double[] run(double[] inputs) {
              // initialize inputs
              for (int i = 0; i &lt; network[0].Length; i++) {
                  network[0][i].activation = inputs[i];
              }
              
              for (int i = 0; i &lt; network.Length - 1; i++) {
                  for (int j = 0; j &lt; network[i].Length; j++) {
                      Neuron neuron = network[i][j];
                      neuron.fire();
                  }
                  for (int j = 0; j &lt; network[i + 1].Length; j++) {
                      Neuron neuron = network[i + 1][j];
                      neuron.activate();
                  }
              }
              
              // get outputs
              
              Neuron[] outputs = network[network.Length - 1];
              double[] activations = new double[outputs.Length];
              
              for (int i = 0; i &lt; outputs.Length; i++) {
                  activations[i] = outputs[i].activation;
              }
              
              return activations;
          }
          
          public double iteration() {
              Random rand = new Random();
              
              int iterationLength = trainingLength/iterations;
              double[] costs = new double[iterationLength];
              
              // training
              
              for (int iter = 0; iter &lt; iterationLength; iter++) {
                  double[] inputs = { (rand.NextDouble() ), ( rand.NextDouble() )};
                  double[] answers = { inputs[0] + inputs[1] };
                  double[] outputs = run(inputs);
                  
                  for (int i = 0; i &lt; network.Length; i++) {
                      for (int j = 0; j &lt; network[i].Length; j++) {
                          Neuron neuron = network[i][j];
                          neuron.desiredBias = new double[iterationLength];
                          if (i == network.Length - 1) continue;
                          neuron.desiredWeights = new double[network[i + 1].Length][];
                          for (int k = 0; k &lt; neuron.desiredWeights.Length; k++) {
                              neuron.desiredWeights[k] = new double[iterationLength];
                          }
                      }
                  }
                  
                  double cost = 0;
                  switch(lossFunction) {
                      case Functions.LossFunction.MSE:
                          cost = Functions.MSE(outputs, answers);
                          break;
                  }
                  
                  costs[iter] = cost;
                  
                  double[] changes = new double[network[network.Length - 1].Length];
                  switch(lossFunction) {
                      case Functions.LossFunction.MSE:
                          changes = Functions.MSEDerivative(outputs, answers); 
                          break;
                  }
                  
                  for (int i = 0; i &lt; changes.Length; i++) {
                      Neuron neuron = network[network.Length - 1][i];
                      double act = neuron.activation;
                      switch(neuron.activationFunction) {
                          case Functions.ActFunction.Sigmoid:
                              changes[i] *= Functions.sigmoidDerivative(Math.Log(act / (1 - act)));
                              break;
                          case Functions.ActFunction.Linear:
                              changes[i] *= Functions.LinearDerivative(act);
                              break;
                      }
                  }
                  
                  for (int i = network.Length - 1; i &gt; 0; i--) {
                      for (int j = 0; j &lt; network[i].Length; j++) {
                          Neuron bNeuron = network[i][j];
                          bNeuron.desiredBias[iter] += changes[j]; // bias partial derivative is always one
                          for (int k = 0; k &lt; network[i - 1].Length; k++) {
                              Neuron aNeuron = network[i - 1][k];
                              aNeuron.desiredWeights[bNeuron.y][iter] += aNeuron.weights[bNeuron.y] * changes[j]; // chain rule
                          }
                      }
                      
                      double[] holderChanges = changes;
                      changes = new double[network[i - 1].Length];
                      for (int j = 0; j &lt; changes.Length; j++) {
                          for (int k = 0; k &lt; holderChanges.Length; k++) {
                              Neuron aNeuron = network[i - 1][j];
                              Neuron bNeuron = network[i][k];
                              switch(aNeuron.activationFunction) {
                                  case Functions.ActFunction.Sigmoid:
                                      double activation = aNeuron.activation;
                                      changes[j] += Functions.sigmoidDerivative(Math.Log(activation / 1 - activation)) * aNeuron.weights[bNeuron.y] * holderChanges[k];
                                      break;
                                  case Functions.ActFunction.Linear:
                                      activation = aNeuron.activation;
                                      changes[j] += Functions.LinearDerivative(activation) * aNeuron.weights[bNeuron.y] * holderChanges[k];
                                      break;
                              }
                          }
                      }
                  }
                  
                  // reset activations 
              
                  for (int i = 0; i &lt; network.Length; i++) {
                      for (int j = 0; j &lt; network[i].Length; j++) {
                          Neuron neuron = network[i][j];
                          neuron.activation = 0;
                      }
                  }
              }
              
              // after everything we now actually add the desired changes
              
              // the weights
              for (int i = 0; i &lt; network.Length - 1; i++) {
                  for (int j = 0; j &lt; network[i].Length; j++) {
                      Neuron neuron = network[i][j];
                      for (int k = 0; k &lt; neuron.desiredWeights.Length; k++) {
                          double weightSum = 0;
                          foreach (double weight in neuron.desiredWeights[k]) {
                              weightSum += weight;
                          }
                          weightSum /= iterationLength;
                          neuron.weights[k] -= learningRate * weightSum;
                      }
                  }
              }
              
              // the biases
              for (int i = 1; i &lt; network.Length; i++) {
                  for (int j = 0; j &lt; network[i].Length; j++) {
                      Neuron neuron = network[i][j];
                      double biasSum = 0;
                      foreach (double bias in neuron.desiredBias) {
                          biasSum += bias;
                      }
                      biasSum /= iterationLength;
                      neuron.bias -= learningRate * biasSum;
                  }
              }
              
              double netcost = 0;
              foreach (double cost in costs) {
                  netcost += cost;
              }
              
              return netcost/costs.Length;
          }
          
          public void epoch(bool showInfo) {
              Console.WriteLine(&quot;Starting training...\n&quot;);
              
              for (int iter = 0; iter &lt; iterations; iter++) {
                  double cost = iteration();
                  if (showInfo) info();
                  
                  Console.WriteLine(&quot;Cost for iteration &quot; + iter + &quot;: &quot; + cost + &quot;\n&quot;);
                  
                  // reset desired
                  
                  for (int k = 0; k &lt; network.Length - 1; k++) {
                      for (int l = 0; l &lt; network[k].Length; l++) {
                          Neuron neuron = network[k][l];
                          for (int i = 0; i &lt; neuron.desiredWeights.Length; i++) {
                              for (int j = 0; j &lt; neuron.desiredWeights[i].Length; j++) {
                                  neuron.desiredWeights[i][j] = 0;
                              }
                          }
                      }
                  }
                  for (int k = 1; k &lt; network.Length; k++) {
                      for (int l = 0; l &lt; network[k].Length; l++) {
                          Neuron neuron = network[k][l];
                          for (int i = 0; i &lt; neuron.desiredBias.Length; i++) {
                              neuron.desiredBias[i] = 0;
                          }
                      }
                  }
              }
              
              Console.WriteLine(&quot;Training finished!\n&quot;);
          }
          
          public void prompt() {
              while(true) {
                  
                  Console.WriteLine(&quot;What would you like to do?&quot;);
                  Console.WriteLine(&quot;Train, info, run or end?&quot;);
                  
                  string answer = Console.ReadLine();
                  
                  if (answer.ToLower() == &quot;end&quot;) {
                      Console.WriteLine(&quot;...&quot;);
                      break;
                  }
                  
                  switch(answer.ToLower()) {
                      case &quot;info&quot;:
                          Console.WriteLine();
                          info();
                          Console.WriteLine();
                          break;
                      case &quot;run&quot;:
                          double[] runanswer = new double[network[0].Length];
                          while (true) {
                              try {
                                  for (int i = 0; i &lt; network[0].Length; i++) {
                                      Console.WriteLine(&quot;Enter input &quot; + i);
                                      runanswer[i] = Convert.ToDouble(Console.ReadLine());
                                  }
                                  
                                  Console.WriteLine(&quot;Running...\n&quot;);
                                  
                                  double[] outputs = run(runanswer);
                                  string _string = &quot;&quot;;
                                  
                                  foreach (double output in outputs) {
                                      _string += output;
                                  }
                                  
                                  Console.WriteLine(&quot;Here were the outputs:&quot;);
                                  Console.WriteLine(_string + &quot;\n&quot;);
                                  break;
                              } catch (FormatException exception) {
                                  Console.WriteLine(&quot;Only numbers!&quot;);
                                  continue;
                              }
                          }
                          break;
                      case &quot;train&quot;:
                          int trainanswer = 0;
                          int trainanswer2 = 0;
                          double trainanswer3 = 0;
                          while (true) {
                              try {
                                  Console.WriteLine(&quot;Enter epoch length:&quot;);
                                  trainanswer = Convert.ToInt32(Console.ReadLine());
                                  trainingLength = trainanswer;
                                  
                                  Console.WriteLine(&quot;\nEnter batch size:&quot;);
                                  trainanswer2 = Convert.ToInt32(Console.ReadLine());
                                  iterations = trainanswer2;
                                  
                                  Console.WriteLine(&quot;\nEnter learning rate:&quot;);
                                  trainanswer3 = Convert.ToDouble(Console.ReadLine());
                                  learningRate = trainanswer3;
                                  
                                  for (int i = 0; i &lt; trainanswer2; i++) {
                                      double cost = iteration();
                                      Console.WriteLine(&quot;The cost was: &quot; + cost + &quot;\n&quot;);
                                  }
                                  break;
                              } catch (FormatException exception) {
                                  Console.WriteLine(&quot;Only numbers!&quot;);
                                  continue;
                              }
                          }
                          break;
                      default:
                          Console.WriteLine(&quot;Invalid command!\n&quot;);
                          break;
                  }
              }
          }
      
      }
      
      public class Run {
          public static void Main(string[] args) {
              
              Network brain = new Network(new int[] { 2, 1 }, .01, 50, 50, Functions.LossFunction.MSE);
              brain.init();
              
              int i = 7;
              while (i &gt; 0) {
                  brain.epoch(false);
                  i--;
              }
              
              brain.prompt();
              
          }
      }
      
  }
  </code></pre>
  <p>*** to clarify if you don't understand my question, I'm just asking if i'm implementing backpropagation correctly</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-05T16:38:59.850Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Tags: <backpropagation><definitions>
Title: Is my neural network working?
UserDisplayName: denvr
ThreadText: # Is my neural network working?
  
  (Asked by: username_0 on 2022-10-05)
  
  I recently just finished programming a neural network in c#, and it seems like it's working. My question is if I'm doing it right. It's a very confusing process so I will explain.
  Basically every neuron in the network has a bias (except the first layer) and fully connected weights to the next. every neuron also has a double[] desiredBias which holds all the desired changes to the neuron for every iteration in a given mini batch. The same thing for the weights. So I start from the last layer, and from there I calculate the "sensitivity" you might say between the cost of a given neuron on that last layer and the bias, and i take note of it (put it into changes). Then for every weight connected to the layer behind, I also calculate the sensitivity for that one. And I keep doing that until I get to the second to first layer. And to take note of the desiredChanges for neurons not in the last layer, I use this array called "changes". For the first layer it's basically just a list of all the derivatives of the activations (z) with respect to the actual activations (sigmoid(z)) respectively. After I shift to the layer behind, and behind, every time I do that, I create a new changes array, which is the size of the layer behind, and for each item in the previous changes array, I take the sum of all of them multiplied by the partial derivative of that same z-activation with respect to the previous layer's z activation. I do this for every neuron in the previous layer.
  after all of that I end up with for every neuron a list of desired changes to it's bias, and a list of lists which contain desired changes to the weights of each neuron. Then for each neuron (except for the first of course) I average together all of the biases and SUBTRACT it from the neuron's bias. I do the same thing for the weights.
  Is this the correct process to go through for exactly one batch of training data?
  here's my code:
  using System;
  
  namespace NeuralNetwork {
      
      public class Functions {
          
          // Activation Functions
          
          public static double sigmoid(double x) {
              return 1 / (1 + Math.Pow(Math.E, -x));
          }
          public static double ReLU(double x) {
              return Math.Max(0, x);
          }
          public static double Tanh(double x) {
              return Math.Tanh(x);
          }
          public static double Linear(double x) {
              return x;
          }
          
          // Derivatives
          
          public static double sigmoidDerivative(double x) {
              return sigmoid(x) * (1 - sigmoid(x));
          }
          public static double ReLUDerivative(double x) {
              return (x > 0) ? 1 : 0;
          }
          public static double TanhDerivative(double x) {
              return 1 - (Math.Pow(Tanh(x) * Tanh(x), 2));
          }
          public static double LinearDerivative(double x) {
              return 1;
          }
          
          public enum ActFunction {
              Sigmoid,
              ReLU,
              Tanh,
              Linear
          }
          
          // Loss Functions
          
          public static double MSE(double[] outputs, double[] desired) {
              double sum = 0;
              for (int i = 0; i < outputs.Length; i++) {
                  sum += Math.Pow((outputs[i] - desired[i]), 2);
              }
              return sum / outputs.Length;
          }
          
          // Derivatives
          
          public static double[] MSEDerivative(double[] outputs, double[] desired) {
              double[] derivatives = new double[outputs.Length];
              for (int i = 0; i < outputs.Length; i++) {
                  derivatives[i] = 2 * (outputs[i] - desired[i]);
              }
              return derivatives;
          }
          
          public enum LossFunction {
              MSE
          }
          
      }
      
      public class Neuron {
      
          public double activation = 0;
          public Functions.ActFunction activationFunction = Functions.ActFunction.Linear;
          
          public double bias;
          public double[] weights = new double[] {};
          public int x;
          public int y;
          public Neuron[][] network;
          
          public double[] desiredBias;
          public double[][] desiredWeights;
          
          public void fire() {
              for (int i = 0; i < network[x + 1].Length; i++) {
                  Neuron next = network[x + 1][i];
                  next.activation += activation * weights[i];
              }
          }
      
          public void activate() {
              double n_activation = activation + bias;
              
              switch(activationFunction) {
                  case Functions.ActFunction.Sigmoid:
                      activation = Functions.sigmoid(n_activation);
                      break;
                  case Functions.ActFunction.ReLU:
                      activation = Functions.ReLU(n_activation);
                      break;
                  case Functions.ActFunction.Tanh:
                      activation = Functions.Tanh(n_activation);
                      break;
                  case Functions.ActFunction.Linear:
                      activation = Functions.Linear(n_activation);
                      break;
              }
          }
          
      }
      
      public class Network {
      
          public Neuron[][] network;
          public double learningRate;
          public int iterations;
          public Functions.LossFunction lossFunction;
          public int trainingLength;
      
          public Network(int[] size, double _learningRate, int _iterations, int _trainingLength, Functions.LossFunction _lossFunction) {
              learningRate = _learningRate;
              iterations = _iterations;
              trainingLength = _trainingLength;
              lossFunction = _lossFunction;
              network = new Neuron[size.Length][];
              for (int i = 0; i < size.Length; i++) {
                  network[i] = new Neuron[size[i]];
              }
              for (int i = 0; i < size.Length; i++) { 
                  for (int j = 0; j < size[i]; j++) { 
                      network[i][j] = new Neuron();
                  }
              }
              for (int i = 0; i < network.Length - 1; i++) {
                  for (int j = 0; j < network[i].Length; j++) { 
                      Neuron neuron = network[i][j];
                      neuron.weights = new double[network[i + 1].Length];
                      neuron.desiredWeights = new double[neuron.weights.Length][]; // i would have put iterations right here but the compiler thinks otherwise
                      for (int k = 0; k < neuron.desiredWeights.Length; k++) {
                          neuron.desiredWeights[k] = new double[trainingLength/iterations];
                      }
                  }
              }
              for (int i = 1; i < network.Length; i++) {
                  for (int j = 0; j < network[i].Length; j++) { 
                      Neuron neuron = network[i][j];
                      neuron.desiredBias = new double[trainingLength/iterations];
                  }
              }
          }
      
          public void init() {
              Random rand = new Random();
              for (int i = 0; i < network.Length; i++) {
                  for (int j = 0; j < network[i].Length; j++) {
                      Neuron neuron = network[i][j];
                      neuron.x = i;
                      neuron.y = j;
                      neuron.network = network;
                      
                      if (i != 0) {
                          neuron.bias = (rand.NextDouble() * 2) - 1;
                      }
                      
                      if (i != network.Length - 1) {
                          for (int k = 0; k < network[neuron.x + 1].Length; k++) {
                              neuron.weights[k] = (rand.NextDouble() * 20) - 1;
                          }
                      }
                  }
              }
          }
          
          public void info() {
              Console.WriteLine("Here are the statistics for the network:\n");
              
              for (int i = 0; i < network.Length; i++) {
                  for (int j = 0; j < network[i].Length; j++) {
                      Neuron neuron = network[i][j];
                      Console.WriteLine("Neuron " + j + " of layer " + i);
                      //Console.WriteLine("Activation: " + neuron.activation);
                      Console.WriteLine("Bias: " + neuron.bias);
                      Console.WriteLine("X and Y: " + neuron.x + ", " + neuron.y);
                      string weights = "Weights: ";
                      foreach (double weight in neuron.weights) {
                          weights += weight + " ";
                      }
                      Console.WriteLine(weights);
                      Console.WriteLine();
                  }
              }
              
              Console.WriteLine();
          }
          
          public double[] run(double[] inputs) {
              // initialize inputs
              for (int i = 0; i < network[0].Length; i++) {
                  network[0][i].activation = inputs[i];
              }
              
              for (int i = 0; i < network.Length - 1; i++) {
                  for (int j = 0; j < network[i].Length; j++) {
                      Neuron neuron = network[i][j];
                      neuron.fire();
                  }
                  for (int j = 0; j < network[i + 1].Length; j++) {
                      Neuron neuron = network[i + 1][j];
                      neuron.activate();
                  }
              }
              
              // get outputs
              
              Neuron[] outputs = network[network.Length - 1];
              double[] activations = new double[outputs.Length];
              
              for (int i = 0; i < outputs.Length; i++) {
                  activations[i] = outputs[i].activation;
              }
              
              return activations;
          }
          
          public double iteration() {
              Random rand = new Random();
              
              int iterationLength = trainingLength/iterations;
              double[] costs = new double[iterationLength];
              
              // training
              
              for (int iter = 0; iter < iterationLength; iter++) {
                  double[] inputs = { (rand.NextDouble() ), ( rand.NextDouble() )};
                  double[] answers = { inputs[0] + inputs[1] };
                  double[] outputs = run(inputs);
                  
                  for (int i = 0; i < network.Length; i++) {
                      for (int j = 0; j < network[i].Length; j++) {
                          Neuron neuron = network[i][j];
                          neuron.desiredBias = new double[iterationLength];
                          if (i == network.Length - 1) continue;
                          neuron.desiredWeights = new double[network[i + 1].Length][];
                          for (int k = 0; k < neuron.desiredWeights.Length; k++) {
                              neuron.desiredWeights[k] = new double[iterationLength];
                          }
                      }
                  }
                  
                  double cost = 0;
                  switch(lossFunction) {
                      case Functions.LossFunction.MSE:
                          cost = Functions.MSE(outputs, answers);
                          break;
                  }
                  
                  costs[iter] = cost;
                  
                  double[] changes = new double[network[network.Length - 1].Length];
                  switch(lossFunction) {
                      case Functions.LossFunction.MSE:
                          changes = Functions.MSEDerivative(outputs, answers); 
                          break;
                  }
                  
                  for (int i = 0; i < changes.Length; i++) {
                      Neuron neuron = network[network.Length - 1][i];
                      double act = neuron.activation;
                      switch(neuron.activationFunction) {
                          case Functions.ActFunction.Sigmoid:
                              changes[i] *= Functions.sigmoidDerivative(Math.Log(act / (1 - act)));
                              break;
                          case Functions.ActFunction.Linear:
                              changes[i] *= Functions.LinearDerivative(act);
                              break;
                      }
                  }
                  
                  for (int i = network.Length - 1; i > 0; i--) {
                      for (int j = 0; j < network[i].Length; j++) {
                          Neuron bNeuron = network[i][j];
                          bNeuron.desiredBias[iter] += changes[j]; // bias partial derivative is always one
                          for (int k = 0; k < network[i - 1].Length; k++) {
                              Neuron aNeuron = network[i - 1][k];
                              aNeuron.desiredWeights[bNeuron.y][iter] += aNeuron.weights[bNeuron.y] * changes[j]; // chain rule
                          }
                      }
                      
                      double[] holderChanges = changes;
                      changes = new double[network[i - 1].Length];
                      for (int j = 0; j < changes.Length; j++) {
                          for (int k = 0; k < holderChanges.Length; k++) {
                              Neuron aNeuron = network[i - 1][j];
                              Neuron bNeuron = network[i][k];
                              switch(aNeuron.activationFunction) {
                                  case Functions.ActFunction.Sigmoid:
                                      double activation = aNeuron.activation;
                                      changes[j] += Functions.sigmoidDerivative(Math.Log(activation / 1 - activation)) * aNeuron.weights[bNeuron.y] * holderChanges[k];
                                      break;
                                  case Functions.ActFunction.Linear:
                                      activation = aNeuron.activation;
                                      changes[j] += Functions.LinearDerivative(activation) * aNeuron.weights[bNeuron.y] * holderChanges[k];
                                      break;
                              }
                          }
                      }
                  }
                  
                  // reset activations 
              
                  for (int i = 0; i < network.Length; i++) {
                      for (int j = 0; j < network[i].Length; j++) {
                          Neuron neuron = network[i][j];
                          neuron.activation = 0;
                      }
                  }
              }
              
              // after everything we now actually add the desired changes
              
              // the weights
              for (int i = 0; i < network.Length - 1; i++) {
                  for (int j = 0; j < network[i].Length; j++) {
                      Neuron neuron = network[i][j];
                      for (int k = 0; k < neuron.desiredWeights.Length; k++) {
                          double weightSum = 0;
                          foreach (double weight in neuron.desiredWeights[k]) {
                              weightSum += weight;
                          }
                          weightSum /= iterationLength;
                          neuron.weights[k] -= learningRate * weightSum;
                      }
                  }
              }
              
              // the biases
              for (int i = 1; i < network.Length; i++) {
                  for (int j = 0; j < network[i].Length; j++) {
                      Neuron neuron = network[i][j];
                      double biasSum = 0;
                      foreach (double bias in neuron.desiredBias) {
                          biasSum += bias;
                      }
                      biasSum /= iterationLength;
                      neuron.bias -= learningRate * biasSum;
                  }
              }
              
              double netcost = 0;
              foreach (double cost in costs) {
                  netcost += cost;
              }
              
              return netcost/costs.Length;
          }
          
          public void epoch(bool showInfo) {
              Console.WriteLine("Starting training...\n");
              
              for (int iter = 0; iter < iterations; iter++) {
                  double cost = iteration();
                  if (showInfo) info();
                  
                  Console.WriteLine("Cost for iteration " + iter + ": " + cost + "\n");
                  
                  // reset desired
                  
                  for (int k = 0; k < network.Length - 1; k++) {
                      for (int l = 0; l < network[k].Length; l++) {
                          Neuron neuron = network[k][l];
                          for (int i = 0; i < neuron.desiredWeights.Length; i++) {
                              for (int j = 0; j < neuron.desiredWeights[i].Length; j++) {
                                  neuron.desiredWeights[i][j] = 0;
                              }
                          }
                      }
                  }
                  for (int k = 1; k < network.Length; k++) {
                      for (int l = 0; l < network[k].Length; l++) {
                          Neuron neuron = network[k][l];
                          for (int i = 0; i < neuron.desiredBias.Length; i++) {
                              neuron.desiredBias[i] = 0;
                          }
                      }
                  }
              }
              
              Console.WriteLine("Training finished!\n");
          }
          
          public void prompt() {
              while(true) {
                  
                  Console.WriteLine("What would you like to do?");
                  Console.WriteLine("Train, info, run or end?");
                  
                  string answer = Console.ReadLine();
                  
                  if (answer.ToLower() == "end") {
                      Console.WriteLine("...");
                      break;
                  }
                  
                  switch(answer.ToLower()) {
                      case "info":
                          Console.WriteLine();
                          info();
                          Console.WriteLine();
                          break;
                      case "run":
                          double[] runanswer = new double[network[0].Length];
                          while (true) {
                              try {
                                  for (int i = 0; i < network[0].Length; i++) {
                                      Console.WriteLine("Enter input " + i);
                                      runanswer[i] = Convert.ToDouble(Console.ReadLine());
                                  }
                                  
                                  Console.WriteLine("Running...\n");
                                  
                                  double[] outputs = run(runanswer);
                                  string _string = "";
                                  
                                  foreach (double output in outputs) {
                                      _string += output;
                                  }
                                  
                                  Console.WriteLine("Here were the outputs:");
                                  Console.WriteLine(_string + "\n");
                                  break;
                              } catch (FormatException exception) {
                                  Console.WriteLine("Only numbers!");
                                  continue;
                              }
                          }
                          break;
                      case "train":
                          int trainanswer = 0;
                          int trainanswer2 = 0;
                          double trainanswer3 = 0;
                          while (true) {
                              try {
                                  Console.WriteLine("Enter epoch length:");
                                  trainanswer = Convert.ToInt32(Console.ReadLine());
                                  trainingLength = trainanswer;
                                  
                                  Console.WriteLine("\nEnter batch size:");
                                  trainanswer2 = Convert.ToInt32(Console.ReadLine());
                                  iterations = trainanswer2;
                                  
                                  Console.WriteLine("\nEnter learning rate:");
                                  trainanswer3 = Convert.ToDouble(Console.ReadLine());
                                  learningRate = trainanswer3;
                                  
                                  for (int i = 0; i < trainanswer2; i++) {
                                      double cost = iteration();
                                      Console.WriteLine("The cost was: " + cost + "\n");
                                  }
                                  break;
                              } catch (FormatException exception) {
                                  Console.WriteLine("Only numbers!");
                                  continue;
                              }
                          }
                          break;
                      default:
                          Console.WriteLine("Invalid command!\n");
                          break;
                  }
              }
          }
      
      }
      
      public class Run {
          public static void Main(string[] args) {
              
              Network brain = new Network(new int[] { 2, 1 }, .01, 50, 50, Functions.LossFunction.MSE);
              brain.init();
              
              int i = 7;
              while (i > 0) {
                  brain.epoch(false);
                  i--;
              }
              
              brain.prompt();
              
          }
      }
      
  }
  
  *** to clarify if you don't understand my question, I'm just asking if i'm implementing backpropagation correctly
  
  **Tags:** backpropagation, definitions
  > **Comments:**
  > **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-10-06. Score: 1 | Accepted answer
  
  Unfortunately I highly doubt anyone is going to check your code or exact implementation as that is a very time consuming process. Luckily for you though, you don't need someone to check, you can check it yourself with **gradient checking**! The basic process is to manually calculate the exact gradient for every single parameter for an input by varying every parameter one at a time, then seeing how each changes your output. With a proper implementation of backpropagation, the gradients will match.
  
  [Here's another question describing this process on stack overflow](https://stackoverflow.com/a/47506918/9546874)
  
  Note though that with ReLU the function is undefined at 0, and you will quickly see how this is a problem when you start doing grad checking. The gradients wont match if you try and check a parameter using the output of a ReLU function when the input to said ReLU is 0. I did some research on this a while ago when I had this problem, and I believe the consensus was to just ignore it
  
  > **Comments:**
  > **username_0**: You're right. I don't know why I thought anyone would. I will try this method, thank you.  
  
  ---
  
  