Answers.Body: <p>Okay - the answer is here  <a href="https://explained.ai/matrix-calculus/#sec6.2" rel="nofollow noreferrer">https://explained.ai/matrix-calculus/#sec6.2</a> and it is pretty involved. Basically, there is a difference when you derive the equation for one neuron and when you have to do practically for a set of neurons. The answer is matrix calculus. Here goes from what I could make out. Feel free to correct if I am wrong</p>
  <h2>Gradient Vector/Matrix/2D tensor of Loss function wrto Weight</h2>
  <p><span class="math-container">$$
  C = \frac{1}{2} \sum_j (y_j-a^L_j)^2
  $$</span></p>
  <p>Assuming a neural net with 2 layers, we have the final Loss as</p>
  <p><span class="math-container">$$
  C = \frac{1}{2} \sum_j (y_j-a^2_j)^2
  $$</span></p>
  <p>Where</p>
  <p><span class="math-container">$$
  a^2 = \sigma(w^2.a^1)
  $$</span></p>
  <p>We can then write</p>
  <p><span class="math-container">$$
  C = \frac{1}{2} \sum_j v^2 \quad \rightarrow (Eq \;A)
  $$</span></p>
  <p>Where</p>
  <p><span class="math-container">$$
  v= y-a^2 
  $$</span></p>
  <h2>Partial Derivative of Loss function wrto Weight</h2>
  <p>For the last layer, lets use Chain Rule to split like below</p>
  <p><span class="math-container">$$
  \frac {\partial C}{\partial w^2} = \frac{\partial v^2}{\partial v} * \frac{\partial v}{\partial w^2} \quad \rightarrow (Eq \;B)
  $$</span></p>
  <p><span class="math-container">$$
   \frac{\partial v^2}{\partial v} =2v \quad \rightarrow (Eq \;B.1)
  $$</span></p>
  <p><span class="math-container">$$
  \frac{\partial v}{\partial w^2}=  \frac{\partial (y-a^2)}{\partial w^2} = 
  0-\frac{\partial a^2}{\partial w^2} \quad \rightarrow (Eq \;B.2)
  $$</span></p>
  <p><span class="math-container">$$
  \frac {\partial C}{\partial w^2} = \frac{1}{2} *2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
  $$</span>
   </p>
  <h3>Now we need to find <span class="math-container">$\frac{\partial a^2}{\partial w^2}$</span></h3>
  <p>Let</p>
  <p><span class="math-container">$$
  a^2= \sigma(sum(w^2 \otimes a^1 )) = \sigma(z^2) 
  $$</span>
  <span class="math-container">$$
  z^2 = sum(w^2 \otimes a^1 )
  $$</span></p>
  <p><span class="math-container">$$
  z^2 = sum(k^2) \; \text {where} \; k^2=w^2 \otimes a^1 
  $$</span></p>
  <p>We now need to derive an intermediate term which we will use later</p>
  <p><span class="math-container">$$
  \frac{\partial z^2}{\partial w^2} =\frac{\partial z^2}{\partial k^2}*\frac{\partial k^2}{\partial w^2}
  $$</span>
  <span class="math-container">$$
  =\frac {\partial sum(k^2)}{\partial k^2}* \frac {\partial (w^2 \otimes a^1 )} {\partial w^2}
  $$</span>
  <span class="math-container">$$
  \frac{\partial z^2}{\partial w^2} = (1^{\rightarrow})^T* diag(a^1) =(a^{1})^T \quad \rightarrow (Eq \;B.3)
  $$</span>
  How the above is, you need to check this  in <a href="https://explained.ai/matrix-calculus/#sec6.2" rel="nofollow noreferrer">https://explained.ai/matrix-calculus/#sec6.2</a></p>
  <p>Basically though these are written like scalar here; actually all these are partial differention of vector by vector, or vector by scalar; and a set of vectors can be represented as the matrix here.</p>
  <p>Note that the vector dot product <span class="math-container">$w.a$</span> when applied on matrices becomes the elementwise multiplication <span class="math-container">$w^2 \otimes a^1$</span> (also called Hadamard product)</p>
  <p>Going back to  <span class="math-container">$Eq \;(B.2)$</span></p>
  <p><span class="math-container">$$
  \frac {\partial a^2}{\partial w^2} = \frac{\partial a^2}{\partial z^2} * \frac{\partial z^2}{\partial w^2}
  $$</span></p>
  <p>Using <span class="math-container">$Eq \;(B.3)$</span> for the term in left</p>
  <p><span class="math-container">$$
  =  \frac{\partial a^2}{\partial z^2} * (a^{1})^T
  $$</span></p>
  <p><span class="math-container">$$
  =  \frac{\partial \sigma(z^2)}{\partial z^2} * (a^{1})^T
  $$</span></p>
  <p><span class="math-container">$$
  \frac {\partial a^2}{\partial w^2} =   \sigma^{'}(z^2) * (a^{1})^T \quad \rightarrow (Eq \;B.4)
  $$</span></p>
  <p>Now lets got back to partial derivative of Loss function wrto to weight</p>
  <p><span class="math-container">$$
  \frac {\partial C}{\partial w^2} = \frac {1}{2}*2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
  $$</span>
  Using <span class="math-container">$Eq \;(B.4)$</span> to substitute in the last term</p>
  <p><span class="math-container">$$
  = v(0- \sigma^{'}(z^2) * (a^{1})^T)
  $$</span></p>
  <p><span class="math-container">$$
  = v*-1*\sigma^{'}(z^2) * (a^{1})^T
  $$</span></p>
  <p><span class="math-container">$$
  = (y-a^2)*-1*\sigma^{'}(z^2) * (a^{1})^T
  $$</span></p>
  <p><span class="math-container">$$
  \frac {\partial C}{\partial w^2}= (a^2-y)*\sigma^{'}(z^2) * (a^{1})^T
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-26T18:25:39.407Z
Answers.LastActivityDate: 2021-03-27T04:32:53.297Z
Answers.LastEditDate: 2021-03-27T04:32:53.297Z
Answers.UserDisplayName: Alex Punnen
Body: <p>I am trying to code a two layered neural network simple NN as I have described here <a href="https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/" rel="nofollow noreferrer">https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/</a></p>
  <p>I am getting stuck on the last step of updating the weights after calculating the gradients for the outer and inner layers via back-propagation</p>
  <pre><code>#---------------------------------------------------------------
  
  # Two layered NW. Using from (1) and the equations we derived as explanations
  # (1) http://iamtrask.github.io/2015/07/12/basic-python-network/
  #---------------------------------------------------------------
  
  import numpy as np
  # seed random numbers to make calculation deterministic 
  np.random.seed(1)
  
  # pretty print numpy array
  np.set_printoptions(formatter={'float': '{: 0.3f}'.format})
  
  # let us code our sigmoid funciton
  def sigmoid(x):
      return 1/(1+np.exp(-x))
  
  # let us add a method that takes the derivative of x as well
  def derv_sigmoid(x):
     return x*(1-x)
  
  # set learning rate as 1 for this toy example
  learningRate =  1
  
  # input x, also used as the training set here
  x = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1]  ])
  
  # desired output for each of the training set above
  y = np.array([[0,1,1,0]]).T
  
  # Explanaiton - as long as input has two ones, but not three, ouput is One
  &quot;&quot;&quot;
  Input [0,0,1]  Output = 0
  Input [0,1,1]  Output = 1
  Input [1,0,1]  Output = 1
  Input [1,1,1]  Output = 0
  &quot;&quot;&quot;
  
  input_rows = 4
  # Randomly initalised weights
  weight1 =  np.random.random((3,input_rows))
  weight2 =  np.random.random((input_rows,1))
  
  print(&quot;Shape weight1&quot;,np.shape(weight1)) #debug
  print(&quot;Shape weight2&quot;,np.shape(weight2)) #debug
  
  # Activation to layer 0 is taken as input x
  a0 = x
  
  iterations = 1000
  for iter in range(0,iterations):
  
    # Forward pass - Straight Forward
    z1= x @ weight1
    a1 = sigmoid(z1) 
    z2= a1 @ weight2
    a2 = sigmoid(z2) 
  
    # Backward Pass - Backpropagation 
    delta2  = (y-a2)
    #---------------------------------------------------------------
    # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
    # Eq (A) ---&gt; dC_dw2 = delta2*derv_sigmoid(z2)
    #---------------------------------------------------------------
  
    dC_dw2  = delta2 * derv_sigmoid(a2)
  
    if iter == 0:
      print(&quot;Shape dC_dw2&quot;,np.shape(dC_dw2)) #debug
    
    #---------------------------------------------------------------
    # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
    # Eq (B)---&gt; dC_dw1 = derv_sigmoid(a1)*delta2*derv_sigmoid(a2)*weight2
    # note  delta2*derv_sigmoid(a2) == dC_dw2 
    # dC_dw1 = derv_sigmoid(a1)*dC_dw2*weight2
    #---------------------------------------------------------------
    
    dC_dw1 =  (np.multiply(dC_dw2,weight2.T)) * derv_sigmoid(a1)
    if iter == 0:
      print(&quot;Shape dC_dw1&quot;,np.shape(dC_dw1)) #debug
    
  
    #---------------------------------------------------------------
    #Gradinent descent
    #---------------------------------------------------------------
   
    #weight2 = weight2 - learningRate*dC_dw2 --&gt; these are what the textbook tells
    #weight1 = weight1 - learningRate*dC_dw1 
  
    weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2) # this is what works
    weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1) 
    
  
  print(&quot;New ouput\n&quot;,a2)
  
  </code></pre>
  <p>Why is</p>
  <pre><code>  weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2)
    weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1) 
  </code></pre>
  <p>done instead of</p>
  <pre><code>  #weight2 = weight2 - learningRate*dC_dw2
    #weight1 = weight1 - learningRate*dC_dw1 
  </code></pre>
  <p>I am not getting the source of the equation of updating the weights by multiplying with the activation of the previous layer</p>
  <p>As per gradient descent, the weight update should be</p>
  <p><span class="math-container">$$
    W^{l}_{new} = W^{l}_{old} - \gamma * \frac{\delta C_0}{\delta w^{l}}
  $$</span></p>
  <p>However, what works in practice is</p>
  <p><span class="math-container">$$
   W^{l}_{new} = W^{l}_{old} - \gamma * \sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}},
  $$</span></p>
  <p>where <span class="math-container">$\gamma$</span> is the learning rate.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-19T12:10:11.247Z
Comments.Text: Have you read something about back-propagation before asking this question? It seems that you're asking us how is back-propagation done because you're asking us to explain a certain implementation.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-19T12:12:59.793Z
Comments.Text: sorry if this is a bit silly; yes - I have derived it and trying to code through (this is what I have done so far https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/)
  Wl−1new=Wl−1old−learningRate∗δC0/δwl−1 - is what I have understood the gradeint descent part
Comments.UserDisplayName: Alex Punnen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-19T12:14:44.890Z
Comments.Text: So, is your question: why do they use `np.dot(a0.T,dC_dw1)` rather than `dC_dw1 ` in the update rule? Maybe you can explain these symbols and maybe the code is not strictly necessary and you can write the equations that they are using in the implementation with latex/mathjax.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-19T12:16:59.237Z
Comments.Text: It may also be a good idea to describe the neural network, the loss function, the task you're solving, etc. Back-propagation can change depending on these parameters.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-20T19:06:32.493Z
Comments.Text: Okay - I guess this is the matrix calculus part - https://explained.ai/matrix-calculus/#sec6.2 need to check this more
Comments.UserDisplayName: Alex Punnen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-21T17:51:06.617Z
Comments.Text: I guess the mistake I did was in doing derivation of sigmoid instead of partial derivation - in partial derivation it is https://math.stackexchange.com/a/2219904/284422 $S(x)(1-S(x)).x and since it is matrix calculus that last term will be transpose
Comments.UserDisplayName: Alex Punnen
Tags: <deep-learning><backpropagation><gradient-descent>
Title: In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$?
UserDisplayName: Alex Punnen
ThreadText: # In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$?
  
  (Asked by: username_0 on 2021-03-19)
  
  I am trying to code a two layered neural network simple NN as I have described here <https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/>
  
  I am getting stuck on the last step of updating the weights after calculating the gradients for the outer and inner layers via back-propagation
  
  ```
  #---------------------------------------------------------------
  
  # Two layered NW. Using from (1) and the equations we derived as explanations
  # (1) http://iamtrask.github.io/2015/07/12/basic-python-network/
  #---------------------------------------------------------------
  
  import numpy as np
  # seed random numbers to make calculation deterministic 
  np.random.seed(1)
  
  # pretty print numpy array
  np.set_printoptions(formatter={'float': '{: 0.3f}'.format})
  
  # let us code our sigmoid funciton
  def sigmoid(x):
      return 1/(1+np.exp(-x))
  
  # let us add a method that takes the derivative of x as well
  def derv_sigmoid(x):
     return x*(1-x)
  
  # set learning rate as 1 for this toy example
  learningRate =  1
  
  # input x, also used as the training set here
  x = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1]  ])
  
  # desired output for each of the training set above
  y = np.array([[0,1,1,0]]).T
  
  # Explanaiton - as long as input has two ones, but not three, ouput is One
  """
  Input [0,0,1]  Output = 0
  Input [0,1,1]  Output = 1
  Input [1,0,1]  Output = 1
  Input [1,1,1]  Output = 0
  """
  
  input_rows = 4
  # Randomly initalised weights
  weight1 =  np.random.random((3,input_rows))
  weight2 =  np.random.random((input_rows,1))
  
  print("Shape weight1",np.shape(weight1)) #debug
  print("Shape weight2",np.shape(weight2)) #debug
  
  # Activation to layer 0 is taken as input x
  a0 = x
  
  iterations = 1000
  for iter in range(0,iterations):
  
    # Forward pass - Straight Forward
    z1= x @ weight1
    a1 = sigmoid(z1) 
    z2= a1 @ weight2
    a2 = sigmoid(z2) 
  
    # Backward Pass - Backpropagation 
    delta2  = (y-a2)
    #---------------------------------------------------------------
    # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
    # Eq (A) ---> dC_dw2 = delta2*derv_sigmoid(z2)
    #---------------------------------------------------------------
  
    dC_dw2  = delta2 * derv_sigmoid(a2)
  
    if iter == 0:
      print("Shape dC_dw2",np.shape(dC_dw2)) #debug
    
    #---------------------------------------------------------------
    # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
    # Eq (B)---> dC_dw1 = derv_sigmoid(a1)*delta2*derv_sigmoid(a2)*weight2
    # note  delta2*derv_sigmoid(a2) == dC_dw2 
    # dC_dw1 = derv_sigmoid(a1)*dC_dw2*weight2
    #---------------------------------------------------------------
    
    dC_dw1 =  (np.multiply(dC_dw2,weight2.T)) * derv_sigmoid(a1)
    if iter == 0:
      print("Shape dC_dw1",np.shape(dC_dw1)) #debug
    
  
    #---------------------------------------------------------------
    #Gradinent descent
    #---------------------------------------------------------------
   
    #weight2 = weight2 - learningRate*dC_dw2 --> these are what the textbook tells
    #weight1 = weight1 - learningRate*dC_dw1 
  
    weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2) # this is what works
    weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1) 
    
  
  print("New ouput\n",a2)
  
  ```
  
  Why is
  
  ```
    weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2)
    weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1) 
  
  ```
  
  done instead of
  
  ```
    #weight2 = weight2 - learningRate*dC_dw2
    #weight1 = weight1 - learningRate*dC_dw1 
  
  ```
  
  I am not getting the source of the equation of updating the weights by multiplying with the activation of the previous layer
  
  As per gradient descent, the weight update should be
  
  $$
  W^{l}\_{new} = W^{l}\_{old} - \gamma \* \frac{\delta C\_0}{\delta w^{l}}
  $$
  
  However, what works in practice is
  
  $$
  W^{l}\_{new} = W^{l}\_{old} - \gamma \* \sigma(z^{l-1})\frac{\delta C\_0}{ \delta w^{l}},
  $$
  
  where $\gamma$ is the learning rate.
  
  **Tags:** deep-learning, backpropagation, gradient-descent
  
  > **Comments:**
  > **username_1**: Have you read something about back-propagation before asking this question? It seems that you're asking us how is back-propagation done because you're asking us to explain a certain implementation.  
  > **username_0**: sorry if this is a bit silly; yes - I have derived it and trying to code through (this is what I have done so far https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/) Wl−1new=Wl−1old−learningRate∗δC0/δwl−1 - is what I have understood the gradeint descent part  
  > **username_1**: So, is your question: why do they use `np.dot(a0.T,dC_dw1)` rather than `dC_dw1 ` in the update rule? Maybe you can explain these symbols and maybe the code is not strictly necessary and you can write the equations that they are using in the implementation with latex/mathjax.  
  > **username_1**: It may also be a good idea to describe the neural network, the loss function, the task you're solving, etc. Back-propagation can change depending on these parameters.  
  > **username_0**: Okay - I guess this is the matrix calculus part - https://explained.ai/matrix-calculus/#sec6.2 need to check this more  
  > **username_0**: I guess the mistake I did was in doing derivation of sigmoid instead of partial derivation - in partial derivation it is https://math.stackexchange.com/a/2219904/284422 $S(x)(1-S(x)).x and since it is matrix calculus that last term will be transpose  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-03-26. Score: 1
  
  Okay - the answer is here <https://explained.ai/matrix-calculus/#sec6.2> and it is pretty involved. Basically, there is a difference when you derive the equation for one neuron and when you have to do practically for a set of neurons. The answer is matrix calculus. Here goes from what I could make out. Feel free to correct if I am wrong
  
  ## Gradient Vector/Matrix/2D tensor of Loss function wrto Weight
  
  $$
  C = \frac{1}{2} \sum\_j (y\_j-a^L\_j)^2
  $$
  
  Assuming a neural net with 2 layers, we have the final Loss as
  
  $$
  C = \frac{1}{2} \sum\_j (y\_j-a^2\_j)^2
  $$
  
  Where
  
  $$
  a^2 = \sigma(w^2.a^1)
  $$
  
  We can then write
  
  $$
  C = \frac{1}{2} \sum\_j v^2 \quad \rightarrow (Eq \;A)
  $$
  
  Where
  
  $$
  v= y-a^2
  $$
  
  ## Partial Derivative of Loss function wrto Weight
  
  For the last layer, lets use Chain Rule to split like below
  
  $$
  \frac {\partial C}{\partial w^2} = \frac{\partial v^2}{\partial v} \* \frac{\partial v}{\partial w^2} \quad \rightarrow (Eq \;B)
  $$
  
  $$
  \frac{\partial v^2}{\partial v} =2v \quad \rightarrow (Eq \;B.1)
  $$
  
  $$
  \frac{\partial v}{\partial w^2}= \frac{\partial (y-a^2)}{\partial w^2} =
  0-\frac{\partial a^2}{\partial w^2} \quad \rightarrow (Eq \;B.2)
  $$
  
  $$
  \frac {\partial C}{\partial w^2} = \frac{1}{2} \*2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
  $$
  
  ### Now we need to find $\frac{\partial a^2}{\partial w^2}$
  
  Let
  
  $$
  a^2= \sigma(sum(w^2 \otimes a^1 )) = \sigma(z^2)
  $$
  $$
  z^2 = sum(w^2 \otimes a^1 )
  $$
  
  $$
  z^2 = sum(k^2) \; \text {where} \; k^2=w^2 \otimes a^1
  $$
  
  We now need to derive an intermediate term which we will use later
  
  $$
  \frac{\partial z^2}{\partial w^2} =\frac{\partial z^2}{\partial k^2}\*\frac{\partial k^2}{\partial w^2}
  $$
  $$
  =\frac {\partial sum(k^2)}{\partial k^2}\* \frac {\partial (w^2 \otimes a^1 )} {\partial w^2}
  $$
  $$
  \frac{\partial z^2}{\partial w^2} = (1^{\rightarrow})^T\* diag(a^1) =(a^{1})^T \quad \rightarrow (Eq \;B.3)
  $$
  How the above is, you need to check this in <https://explained.ai/matrix-calculus/#sec6.2>
  
  Basically though these are written like scalar here; actually all these are partial differention of vector by vector, or vector by scalar; and a set of vectors can be represented as the matrix here.
  
  Note that the vector dot product $w.a$ when applied on matrices becomes the elementwise multiplication $w^2 \otimes a^1$ (also called Hadamard product)
  
  Going back to $Eq \;(B.2)$
  
  $$
  \frac {\partial a^2}{\partial w^2} = \frac{\partial a^2}{\partial z^2} \* \frac{\partial z^2}{\partial w^2}
  $$
  
  Using $Eq \;(B.3)$ for the term in left
  
  $$
  = \frac{\partial a^2}{\partial z^2} \* (a^{1})^T
  $$
  
  $$
  = \frac{\partial \sigma(z^2)}{\partial z^2} \* (a^{1})^T
  $$
  
  $$
  \frac {\partial a^2}{\partial w^2} = \sigma^{'}(z^2) \* (a^{1})^T \quad \rightarrow (Eq \;B.4)
  $$
  
  Now lets got back to partial derivative of Loss function wrto to weight
  
  $$
  \frac {\partial C}{\partial w^2} = \frac {1}{2}\*2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
  $$
  Using $Eq \;(B.4)$ to substitute in the last term
  
  $$
  = v(0- \sigma^{'}(z^2) \* (a^{1})^T)
  $$
  
  $$
  = v\*-1\*\sigma^{'}(z^2) \* (a^{1})^T
  $$
  
  $$
  = (y-a^2)\*-1\*\sigma^{'}(z^2) \* (a^{1})^T
  $$
  
  $$
  \frac {\partial C}{\partial w^2}= (a^2-y)\*\sigma^{'}(z^2) \* (a^{1})^T
  $$
  
  
  ---
  
  