Answers.Body: <p>The terms &quot;error&quot;, &quot;delta&quot; and &quot;gradient&quot; in neural network back-propagation are often used as shorthand, or loose explanations for the same thing. This is not strictly correct in all cases, and you should be careful to check the code in context. However, these words do make for suitable variable names, so you will see them in implenentations.</p>
  <p>Back-propagation is repeated application of the <a href="https://www.khanacademy.org/math/ap-calculus-ab/ab-differentiation-2-new/ab-3-1a/a/chain-rule-review" rel="nofollow noreferrer">chain rule of calculus</a>, which can be written:</p>
  <p><span class="math-container">$$\frac{d}{dx} f(g(x)) = \frac{df}{dg} \frac{dg}{dx}$$</span></p>
  <p>This is useful for neural networks, because each layer (and each activation function) is applied as a function to each previous layer, so a neural network is effectively a giant composite of much simpler functions, each applied to the output of the previous one.</p>
  <p>The goal of back-propagation is to figure out the <em>gradient of the loss function with respect to the neural network's changeable parameters</em> (the weights and biases usually). So &quot;gradient&quot; is kind of shorthand for this, although you work it out in part, plus some are temporary useful values that are not part of the final array of partial derivatives.</p>
  <p>Some people like to think of the gradient as an &quot;error correction signal&quot;, and also with careful design, the initial gradient of the loss for the neuron output with respect to the output layer, before the activation is applied, can be <span class="math-container">$\hat{y} - y$</span>, the difference between the current output and the ground truth. So it even looks a little bit like an error value, even though it is a different kind of quantity which happens to equal the error value.</p>
  <p>I am not sure, but I think &quot;delta&quot; is a direct reference to the Greek letter delta in the partial derivative <span class="math-container">$\frac{\delta L}{\delta W_i}$</span>, as the full gradient is an array of these partial derivatives of the loss function with respect to each parameter in turn.</p>
  <p>A good way to understand and isolate yourself from all the different names used in implementation code is to learn the mathematical representation of backpropagation. The maths gives you a less ambiguous view of what is going on, and doesn't use any of those three terms.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-04T21:18:16.120Z
Answers.LastActivityDate: 2023-11-04T21:31:39.543Z
Answers.LastEditDate: 2023-11-04T21:31:39.543Z
Answers.UserDisplayName: Neil Slater
Body: <p><strong>What is the difference between &quot;delta&quot;, &quot;gradient&quot; and &quot;error&quot;,
  are these names the same thing?</strong></p>
  <p>I'm confused because someone once told me that both the names &quot;delta&quot; and &quot;error&quot; are commonly used.
  And I searched the internet, and some explanations also use the name &quot;gradient&quot;.</p>
  <p>For example, this article: <a href="https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/" rel="nofollow noreferrer">https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/</a>, that teaches you how to create a multilayer perceptron neural network in Python, uses the name &quot;delta&quot;,
  but don't call it &quot;gradient&quot;.</p>
  <p>But, this other article: <a href="https://home.agh.edu.pl/%7Evlsi/AI/backp_t_en/backprop.html" rel="nofollow noreferrer">https://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html</a>, it does not use the name &quot;delta&quot; or &quot;gradient&quot;.</p>
  <p><strong>Please, could someone help me understand about &quot;error&quot;, &quot;delta&quot; and &quot;gradient&quot;, what is the difference between these names? are they the same thing?</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-04T21:25:21.993Z
Comments.Text: The second diagram doesn't use the word "delta", but all the values it calculates are called $\delta_{n}$ using the Greek lowercase delta, even whilst it calls it "error"
Comments.UserDisplayName: Neil Slater
Tags: <backpropagation><gradient>
Title: In multilayer perceptron neural networks, are the names "delta", "gradient" and "error" all the same thing? or not?
UserDisplayName: will The J
ThreadText: # In multilayer perceptron neural networks, are the names "delta", "gradient" and "error" all the same thing? or not?
  
  (Asked by: username_0 on 2023-11-02)
  
  **What is the difference between "delta", "gradient" and "error",
  are these names the same thing?**
  
  I'm confused because someone once told me that both the names "delta" and "error" are commonly used.
  And I searched the internet, and some explanations also use the name "gradient".
  
  For example, this article: <https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/>, that teaches you how to create a multilayer perceptron neural network in Python, uses the name "delta",
  but don't call it "gradient".
  
  But, this other article: [https://home.agh.edu.pl/~vlsi/AI/backp\_t\_en/backprop.html](https://home.agh.edu.pl/%7Evlsi/AI/backp_t_en/backprop.html), it does not use the name "delta" or "gradient".
  
  **Please, could someone help me understand about "error", "delta" and "gradient", what is the difference between these names? are they the same thing?**
  
  **Tags:** backpropagation, gradient
  
  > **Comments:**
  > **username_1**: The second diagram doesn't use the word "delta", but all the values it calculates are called $\delta_{n}$ using the Greek lowercase delta, even whilst it calls it "error"  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-04. Score: 2 | Accepted answer
  
  The terms "error", "delta" and "gradient" in neural network back-propagation are often used as shorthand, or loose explanations for the same thing. This is not strictly correct in all cases, and you should be careful to check the code in context. However, these words do make for suitable variable names, so you username_0 see them in implenentations.
  
  Back-propagation is repeated application of the [chain rule of calculus](https://www.khanacademy.org/math/ap-calculus-ab/ab-differentiation-2-new/ab-3-1a/a/chain-rule-review), which can be written:
  
  $$\frac{d}{dx} f(g(x)) = \frac{df}{dg} \frac{dg}{dx}$$
  
  This is useful for neural networks, because each layer (and each activation function) is applied as a function to each previous layer, so a neural network is effectively a giant composite of much simpler functions, each applied to the output of the previous one.
  
  The goal of back-propagation is to figure out the *gradient of the loss function with respect to the neural network's changeable parameters* (the weights and biases usually). So "gradient" is kind of shorthand for this, although you work it out in part, plus some are temporary useful values that are not part of the final array of partial derivatives.
  
  Some people like to think of the gradient as an "error correction signal", and also with careful design, the initial gradient of the loss for the neuron output with respect to the output layer, before the activation is applied, can be $\hat{y} - y$, the difference between the current output and the ground truth. So it even looks a little bit like an error value, even though it is a different kind of quantity which happens to equal the error value.
  
  I am not sure, but I think "delta" is a direct reference to the Greek letter delta in the partial derivative $\frac{\delta L}{\delta W\_i}$, as the full gradient is an array of these partial derivatives of the loss function with respect to each parameter in turn.
  
  A good way to understand and isolate yourself from all the different names used in implementation code is to learn the mathematical representation of backpropagation. The maths gives you a less ambiguous view of what is going on, and doesn't use any of those three terms.
  
  
  ---
  
  