Answers.Body: <p>Your last article is mainly focused on programming the classic full GD BP algo from scratch using pure Python under the default MSE cost function but never mentioned or emphasized in the main body except somewhere in the extremely long comment section, thus it may cause you cannot link with chain rule.</p>
  <p>The first outer chain under MSE is obviously just the error signal from the final output compared with its true target, the 2nd chain is the derivative of the activation function which is the classic sigmoid function which you're already aware, and the final inner chain is the output of the originating neuron of the updated weight edge in the previous layer or just one of the input signals if there's none. These correspond to the three usual multiplicative terms in a delta rule.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-10T00:35:09.853Z
Answers.Text: Thanks a lot for explaining better the chain rule!
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-09T23:43:32.930Z
Answers.LastActivityDate: 2023-11-10T00:48:36.790Z
Answers.LastEditDate: 2023-11-10T00:48:36.790Z
Answers.UserDisplayName: cinch
Answers.Body: <p>Unless the derivation has been added in a comment, when you read code implementations of backpropagation, then the chain rule has already been used to derive the update rules. As it is basically a rule about multiplying one simpler derivative by another to resolve the derivative of a more complex function, then typically some of the terms in each statement represent <span class="math-container">$\frac{\partial f}{\partial x}$</span> and some represent <span class="math-container">$\frac{\partial g}{\partial f}$</span>. When those are multiplied together you find the numerical value of <span class="math-container">$\frac{\partial g}{\partial x}$</span> from the composite <span class="math-container">$g(f(x))$</span>.</p>
  <p>The &quot;deltas&quot; being stored in variables are components of the gradient of the loss function with respect to the network's internal states and free parameters, at its current value (due to current item, or an aggregate across the current batch).</p>
  <ul>
  <li><p>Some are temporary calculations that don't need to be stored longer term because they are not gradient components of changeable parameters.</p>
  </li>
  <li><p>Some represent components of <span class="math-container">$\nabla_{W,b} J$</span>, the gradient of the cost function <span class="math-container">$J$</span> with respect to the weights and biases of the
  neural network - these are the values that you can change as part of
  training, so they are usually stored in a structure the same shape as the weights and biases so that they can be aggregated across multiple examples and subtracted from <span class="math-container">$W,b$</span> later.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-09T22:18:06.837Z
Answers.Text: Thanks. Please, I have a question: the article uses the Sigmoid, and it shows its derivative.  So according to your explanation in "[...] when you read code implementations of backpropagation, then the chain rule has already been used to derive the update rules.", this means that if I have an activation function complex, and I don't know the derivative of it, so that's where I would apply the derivation to find the derivative?  But in article this derivation has already been done (the derivative of the Sigmoid), I don't need to do the derivation because it was done. Did I understand correctly?
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-09T22:27:51.800Z
Answers.Text: in other words, do I just use the chain rule to find the derivative of the activation function when I don't know what the derivative ?
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-10T14:14:13.497Z
Answers.Text: @willTheJ If you are being shown working code for training, you should not need to do any extra maths on the code or data in order to use it. Quoting the chain rule is explaining *how* it works, and how you might build your own NN from scratch. If you *are* building your own NN from scratch, using e.g. NumPy, and stuck, then please as a new question about that.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-10T17:38:00.510Z
Answers.Text: Okay, I will create new questions for other questions.. Thanks for explaining
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-09T09:13:41.683Z
Answers.LastActivityDate: 2023-11-09T11:08:58.030Z
Answers.LastEditDate: 2023-11-09T11:08:58.030Z
Answers.UserDisplayName: Neil Slater
Body: <p><strong>References:</strong></p>
  <ul>
  <li><strong>Chain rule in Wikipedia:</strong> <a href="https://en.wikipedia.org/wiki/Chain_rule" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Chain_rule</a></li>
  <li><strong>Chain rule in Towards Data Science:</strong> <a href="https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd#:%7E:text=The%20algorithm%20is%20used%20to,parameters%20(weights%20and%20biases)" rel="nofollow noreferrer">https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd#:~:text=The%20algorithm%20is%20used%20to,parameters%20(weights%20and%20biases)</a>.</li>
  </ul>
  <p>Searching the internet in some articles, I learned that the chain rule is used so that the network can identify the contribution of each weight to the error, and how much each weight needs to be adjusted.</p>
  <p>This article I mentioned in the question: <a href="https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/" rel="nofollow noreferrer">https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/</a>, they teaches you how to program a multilayer perceptron neural network from scratch in Python. During the calculations in the backpropagation phase, he uses some different terms such &quot;delta&quot;, and
  it calculates the &quot;delta&quot; layer by layer. I know that the explanation in the article must make use of the chain rule, but I couldn't understand where.</p>
  <p>I couldn't see where the chain rule is. Does this article make use of the chain rule? And where?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-09T00:38:55.487Z
Comments.Text: The chain rule is spread in the 3 core functions of your reference: forward_propagate(network, row), backward_propagate_error(network, expected), update_weights(network, row, l_rate) which are linked in the train_network(network, train, l_rate, n_epoch, n_outputs) function. This is mostly a programming question.
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-09T09:01:03.333Z
Comments.Text: Can you please don't use links in the title and don't use some much bold?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-09T10:38:09.207Z
Comments.Text: @nbro Okay.  Today I will edit the question and remove the title link and remove the bold text.
Comments.UserDisplayName: will The J
Tags: <backpropagation>
Title: Does this article make use of the chain rule? And where?
UserDisplayName: will The J
ThreadText: # Does this article make use of the chain rule? And where?
  
  (Asked by: username_0 on 2023-11-08)
  
  **References:**
  
  * **Chain rule in Wikipedia:** <https://en.wikipedia.org/wiki/Chain_rule>
  * **Chain rule in Towards Data Science:** [https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd#:~:text=The%20algorithm%20is%20used%20to,parameters%20(weights%20and%20biases)](https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd#:%7E:text=The%20algorithm%20is%20used%20to,parameters%20(weights%20and%20biases)).
  
  Searching the internet in some articles, I learned that the chain rule is used so that the network can identify the contribution of each weight to the error, and how much each weight needs to be adjusted.
  
  This article I mentioned in the question: <https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/>, they teaches you how to program a multilayer perceptron neural network from scratch in Python. During the calculations in the backpropagation phase, he uses some different terms such "delta", and
  it calculates the "delta" layer by layer. I know that the explanation in the article must make use of the chain rule, but I couldn't understand where.
  
  I couldn't see where the chain rule is. Does this article make use of the chain rule? And where?
  
  **Tags:** backpropagation
  
  > **Comments:**
  > **username_1**: The chain rule is spread in the 3 core functions of your reference: forward_propagate(network, row), backward_propagate_error(network, expected), update_weights(network, row, l_rate) which are linked in the train_network(network, train, l_rate, n_epoch, n_outputs) function. This is mostly a programming question.  
  > **username_2**: Can you please don't use links in the title and don't use some much bold?  
  > **username_0**: @username_2 Okay.  Today I username_0 edit the question and remove the title link and remove the bold text.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2023-11-09. Score: 2 | Accepted answer
  
  Unless the derivation has been added in a comment, when you read code implementations of backpropagation, then the chain rule has already been used to derive the update rules. As it is basically a rule about multiplying one simpler derivative by another to resolve the derivative of a more complex function, then typically some of the terms in each statement represent $\frac{\partial f}{\partial x}$ and some represent $\frac{\partial g}{\partial f}$. When those are multiplied together you find the numerical value of $\frac{\partial g}{\partial x}$ from the composite $g(f(x))$.
  
  The "deltas" being stored in variables are components of the gradient of the loss function with respect to the network's internal states and free parameters, at its current value (due to current item, or an aggregate across the current batch).
  
  * Some are temporary calculations that don't need to be stored longer term because they are not gradient components of changeable parameters.
  * Some represent components of $\nabla\_{W,b} J$, the gradient of the cost function $J$ with respect to the weights and biases of the
    neural network - these are the values that you can change as part of
    training, so they are usually stored in a structure the same shape as the weights and biases so that they can be aggregated across multiple examples and subtracted from $W,b$ later.
  
  > **Comments:**
  > **username_0**: Thanks. Please, I have a question: the article uses the Sigmoid, and it shows its derivative.  So according to your explanation in "[...] when you read code implementations of backpropagation, then the chain rule has already been used to derive the update rules.", this means that if I have an activation function complex, and I don't know the derivative of it, so that's where I would apply the derivation to find the derivative?  But in article this derivation has already been done (the derivative of the Sigmoid), I don't need to do the derivation because it was done. Did I understand correctly?  
  > **username_0**: in other words, do I just use the chain rule to find the derivative of the activation function when I don't know what the derivative ?  
  > **username_3**: @username_0 If you are being shown working code for training, you should not need to do any extra maths on the code or data in order to use it. Quoting the chain rule is explaining *how* it works, and how you might build your own NN from scratch. If you *are* building your own NN from scratch, using e.g. NumPy, and stuck, then please as a new question about that.  
  > **username_0**: Okay, I username_0 create new questions for other questions.. Thanks for explaining  
  
  ---
  
  ### Answer by username_1 on 2023-11-09. Score: 1
  
  Your last article is mainly focused on programming the classic full GD BP algo from scratch using pure Python under the default MSE cost function but never mentioned or emphasized in the main body except somewhere in the extremely long comment section, thus it may cause you cannot link with chain rule.
  
  The first outer chain under MSE is obviously just the error signal from the final output compared with its true target, the 2nd chain is the derivative of the activation function which is the classic sigmoid function which you're already aware, and the final inner chain is the output of the originating neuron of the updated weight edge in the previous layer or just one of the input signals if there's none. These correspond to the three usual multiplicative terms in a delta rule.
  
  > **Comments:**
  > **username_0**: Thanks a lot for explaining better the chain rule!  
  
  ---
  
  