PART 2 : WRITING A DYNAMIC NEURAL NETWORK FRAMEWORK FROM SCRATCH IN PYTHON
Back Prop Step

Just like the feedforward step we will discuss the helper function that helps us to make the backprop calculations, here instead of the Sigmoid we will use the derivative of the Sigmoid function.

It is formed by multiplying the sigmoid function with the value of 1 minus itself. Its execution in code is :

Now, we come to the backprop part where we use the chain rule to find out the change in the Loss with respect to the Z matrix, this will be the slope of “Z” w.r.t. “J” and we will represent this as “dZ”.

This value of dZ is further used to ding the change in “W” and “b” w.r.t. to “J”, so instead of calculating the change all the way back to J, we and directly use “dZ”. These slopes of Weights and Biases are represented as “dW” and “db”.

As the last layer is followed by the Loss function, the equation of its derivative is a little uniques. Hence, the value of the dZ of the last layer is calculated separately, whereas all the other dZ are calculated within a loop.

After calculating all the dZ, we move on to updating the values of our weights using dW. To get dW we multiply dZ with A of the last layer, then when we have dW we multiply it by our learning rate (alpha) and subtract it from our earlier W to get the new W.

The calculation of the last dW is done differently as we don't have an A0 (i.e. activation in the first input layer)

Similarly, for biases we directly multiply the dZ value by the alpha and subtract it from the old bias to get the new bias, this is done in a loop.
['Angad Sandhu']
['Neural Networks', 'Python', 'AI', 'Framework', 'Machine Learning']