Answers.Body: <p>Backpropagation isn't too much more complicated, but understanding it well will require a bit of mathematics.</p>
  <p><a href="https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/" rel="nofollow noreferrer">This tutorial</a> is my go-to resource when students want more detail, because it includes fully worked through examples.</p>
  <p>Chapter 18 of <a href="https://rads.stackoverflow.com/amzn/click/com/0136042597" rel="nofollow noreferrer" rel="nofollow noreferrer">Russell &amp; Norvig's</a> book includes pseudocode for this algorithm, as well as a derivation, but without good examples.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T20:41:46.657Z
Answers.Text: [Table of Contents for AI: A Modern Approach](http://aima.cs.berkeley.edu/contents.html)
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-13T15:05:05.983Z
Answers.LastActivityDate: 2020-12-22T23:56:17.580Z
Answers.LastEditDate: 2020-12-22T23:56:17.580Z
Answers.UserDisplayName: John Doucette
Answers.Body: <p>Nowadays, there are many resources that cover the back-propagation algorithm and some of them provide step-by-step examples.</p>
  <p>However, in addition to <a href="https://ai.stackexchange.com/a/7545/2444">the other answer</a>, I would like to mention the online book <a href="http://neuralnetworksanddeeplearning.com/" rel="nofollow noreferrer">Neural Networks and Deep Learning</a> by Nielsen that covers <a href="http://neuralnetworksanddeeplearning.com/chap2.html" rel="nofollow noreferrer">the back-propagation algorithm</a> (and other topics) in detail and, at the same, intuitively, although some could disagree. You can find the associated source code <a href="https://github.com/mnielsen/neural-networks-and-deep-learning" rel="nofollow noreferrer">here</a> (which I had consulted a few years ago when I was learning about the topic).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-23T00:10:20.207Z
Answers.LastActivityDate: 2020-12-23T00:10:20.207Z
Answers.UserDisplayName: nbro
Body: <p>I've been wanting to make my own Neural Network in Python, in order to better understand how it works. I've been following <a href="https://www.youtube.com/watch?v=ZzWaow1Rvho" rel="nofollow noreferrer">this</a> series of videos as a sort of guide, but it seems the backpropagation will get much more difficult when you use a larger network, which I plan to do. He doesn't really explain how to scale it to larger ones.</p>
  <p>Currently, my network feeds forward, but I don't have much of an idea of where to start with backpropagation. My code is posted below, to show you where I'm currently at (I'm not asking for coding help, just for some pointers to good sources, and I figure knowing where I'm currently at might help):</p>
  <pre><code>import numpy
  
  class NN:
      prediction = []
      def __init__(self,input_length):
          self.layers = []
          self.input_length = input_length
      def addLayer(self, layer):
          self.layers.append(layer)
          if len(self.layers) &gt;1:
              self.layers[len(self.layers)-1].setWeights(len(self.layers[len(self.layers)-2].neurons))
          else:
              self.layers[0].setWeights(self.input_length)
      def feedForward(self, inputs):
          _inputs = inputs
          for i in range(len(self.layers)):
              self.layers[i].process(_inputs)
              _inputs = self.layers[i].output
          self.prediction = _inputs
  
      def calculateErr(self, target):
          out = []
          for i in range(0,len(self.prediction)):
              out.append(  (self.prediction[i] - target[i]) ** 2  )
          return out
          
  
  class Layer:
  
      neurons = []
      weights = []
      biases = []
      output = []
      
      def __init__(self,length,function):
          for i in range(0,length):
              self.neurons.append(Neuron(function))
              self.biases.append(numpy.random.randn())
  
      def setWeights(self, inlength):
          for i in range(0,inlength):
              self.weights.append([])
              for j in range(0, inlength):
                  self.weights[i].append(numpy.random.randn())
      
      def process(self,inputs):
          for i in range(0, len(self.neurons)):
              self.output.append(self.neurons[i].run(inputs,self.weights[i], self.biases[i]))
      
  
  class Neuron:
      output = 0
      def __init__(self, function):
          self.function = function
      def run(self, inputs, weights, bias):
          self.output = self.function(inputs,weights,bias)
          return self.output
  
  def sigmoid(n):
      return 1/(1+numpy.exp(n))
  
  
  def inputlayer_func(inputs,weights,bias):
      return inputs
  
  def l2_func(inputs,weights,bias):
      out = 0
      
      for i in range(0,len(inputs)):
          out += weights[i] * inputs[i]
      out += bias
      
      return sigmoid(out)
  
  NNet = NN(2)
  
  
  l2 = Layer(1,l2_func)
  
  
  NNet.addLayer(l2)
  NNet.feedForward([2.0,1.0])
  print(NNet.prediction)
  </code></pre>
  <p>So, is there any resource that explains how to implement the back-propagation algorithm step-by-step?</p>
  
Tags: <neural-networks><reference-request><backpropagation><implementation><resource-request>
Title: How can I implement back-propagation for medium-sized neural networks?
UserDisplayName: Caleb H.
ThreadText: # How can I implement back-propagation for medium-sized neural networks?
  
  (Asked by: username_0 on 2018-08-11)
  
  I've been wanting to make my own Neural Network in Python, in order to better understand how it works. I've been following [this](https://www.youtube.com/watch?v=ZzWaow1Rvho) series of videos as a sort of guide, but it seems the backpropagation will get much more difficult when you use a larger network, which I plan to do. He doesn't really explain how to scale it to larger ones.
  
  Currently, my network feeds forward, but I don't have much of an idea of where to start with backpropagation. My code is posted below, to show you where I'm currently at (I'm not asking for coding help, just for some pointers to good sources, and I figure knowing where I'm currently at might help):
  
  ```
  import numpy
  
  class NN:
      prediction = []
      def __init__(self,input_length):
          self.layers = []
          self.input_length = input_length
      def addLayer(self, layer):
          self.layers.append(layer)
          if len(self.layers) >1:
              self.layers[len(self.layers)-1].setWeights(len(self.layers[len(self.layers)-2].neurons))
          else:
              self.layers[0].setWeights(self.input_length)
      def feedForward(self, inputs):
          _inputs = inputs
          for i in range(len(self.layers)):
              self.layers[i].process(_inputs)
              _inputs = self.layers[i].output
          self.prediction = _inputs
  
      def calculateErr(self, target):
          out = []
          for i in range(0,len(self.prediction)):
              out.append(  (self.prediction[i] - target[i]) ** 2  )
          return out
          
  
  class Layer:
  
      neurons = []
      weights = []
      biases = []
      output = []
      
      def __init__(self,length,function):
          for i in range(0,length):
              self.neurons.append(Neuron(function))
              self.biases.append(numpy.random.randn())
  
      def setWeights(self, inlength):
          for i in range(0,inlength):
              self.weights.append([])
              for j in range(0, inlength):
                  self.weights[i].append(numpy.random.randn())
      
      def process(self,inputs):
          for i in range(0, len(self.neurons)):
              self.output.append(self.neurons[i].run(inputs,self.weights[i], self.biases[i]))
      
  
  class Neuron:
      output = 0
      def __init__(self, function):
          self.function = function
      def run(self, inputs, weights, bias):
          self.output = self.function(inputs,weights,bias)
          return self.output
  
  def sigmoid(n):
      return 1/(1+numpy.exp(n))
  
  def inputlayer_func(inputs,weights,bias):
      return inputs
  
  def l2_func(inputs,weights,bias):
      out = 0
      
      for i in range(0,len(inputs)):
          out += weights[i] * inputs[i]
      out += bias
      
      return sigmoid(out)
  
  NNet = NN(2)
  
  l2 = Layer(1,l2_func)
  
  NNet.addLayer(l2)
  NNet.feedForward([2.0,1.0])
  print(NNet.prediction)
  
  ```
  
  So, is there any resource that explains how to implement the back-propagation algorithm step-by-step?
  
  **Tags:** neural-networks, reference-request, backpropagation, implementation, resource-request
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-08-13. Score: 2 | Accepted answer
  
  Backpropagation isn't too much more complicated, but understanding it well will require a bit of mathematics.
  
  [This tutorial](https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/) is my go-to resource when students want more detail, because it includes fully worked through examples.
  
  Chapter 18 of [Russell & Norvig's](https://rads.stackoverflow.com/amzn/click/com/0136042597) book includes pseudocode for this algorithm, as well as a derivation, but without good examples.
  
  > **Comments:**
  > **username_2**: [Table of Contents for AI: A Modern Approach](http://aima.cs.berkeley.edu/contents.html)  
  
  ---
  
  ### Answer by username_3 on 2020-12-23. Score: 0
  
  Nowadays, there are many resources that cover the back-propagation algorithm and some of them provide step-by-step examples.
  
  However, in addition to [the other answer](https://ai.stackexchange.com/a/7545/2444), I would like to mention the online book [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/) by Nielsen that covers [the back-propagation algorithm](http://neuralnetworksanddeeplearning.com/chap2.html) (and other topics) in detail and, at the same, intuitively, although some could disagree. You can find the associated source code [here](https://github.com/mnielsen/neural-networks-and-deep-learning) (which I had consulted a few years ago when I was learning about the topic).
  
  
  ---
  
  