Answers.Body: <blockquote>
  <p>Why is it called back-propagation?</p>
  </blockquote>
  <p>I don't think there is anything special here!</p>
  <p>It's called back-propagation (BP) because, after the forward pass, you compute the partial derivative of the loss function with respect to the parameters of the network, which, in the usual diagrams of a neural network, are placed <strong>before</strong> the output of the network (i.e. to the left of the output if the output of the network is on the right, or to the right if the output of the network is on the left).</p>
  <p>It's also called BP because it is just the application of the <a href="https://en.wikipedia.org/wiki/Chain_rule" rel="nofollow noreferrer">chain rule</a>. Why is this interesting?</p>
  <p>Let me answer this question with an example. Consider the function <span class="math-container">$y=e^{\sin(x^{2})}$</span>. This is a <a href="https://en.wikipedia.org/wiki/Function_composition" rel="nofollow noreferrer">composite function</a>, i.e. a function composed of multiple simpler functions, which, in this case, are <span class="math-container">$e^x$</span>, <span class="math-container">$\sin(x)$</span>, <span class="math-container">$x^2$</span> and <span class="math-container">$x$</span>. To compute the derivative of <span class="math-container">$y$</span> with respect to <span class="math-container">$x$</span>, let's define the following variables</p>
  <p><span class="math-container">\begin{align}
  y &amp;= f(u) = e^u,\\
  u &amp;= g(v) = \sin v = \sin(x^2),\\
  v &amp;= h(x) = x^2
  \end{align}</span></p>
  <p>The derivative of <span class="math-container">$y$</span> with respect to the variable <span class="math-container">$x$</span> is (according to the <a href="https://en.wikipedia.org/wiki/Chain_rule" rel="nofollow noreferrer">chain rule</a>)</p>
  <p><span class="math-container">$$
  \underset{\color{red}{\LARGE \rightarrow}}{
  \frac{dy}{dx} = \frac{dy}{du} \color{green}{\cdot} \frac{du}{dv}  \color{green}{\cdot} 
  \frac{dv}{dx}}
  $$</span></p>
  <p>If you read this equation from the left to the right, you can see that we are going backward (i.e. from the function <span class="math-container">$y$</span> to the function <span class="math-container">$v$</span>). This is the same thing with BP!</p>
  <p>Why is it called &quot;chain rule&quot;? Because you are <a href="https://en.wikipedia.org/wiki/Chain" rel="nofollow noreferrer">chaining</a> different partial derivatives. More specifically, you are multiplying them.</p>
  <p>BP is also known as the <strong>reverse mode of automatic differentiation</strong>. Why? The <a href="https://en.wikipedia.org/wiki/Automatic_differentiation" rel="nofollow noreferrer">automatic differentiation</a> should be self-explanatory, given that the BP algorithm is just the computation of partial derivatives, and you do this <em>automatically</em>, i.e. with a program, rather than by hand. The expression &quot;reverse mode&quot; refers to the fact that we compute the derivatives from the outer function (which, in the example above, is <span class="math-container">$e^x$</span>) to the inner function (which, in the example above, is <span class="math-container">$x$</span>). <a href="https://en.wikipedia.org/wiki/Automatic_differentiation" rel="nofollow noreferrer">The Wikipedia article related to automatic differentiation</a> provides more details.</p>
  <blockquote>
  <p>What exactly are you back-propagating?</p>
  </blockquote>
  <p>The partial derivative of the loss function <span class="math-container">$\mathcal{L}$</span> with respect to a parameter <span class="math-container">$w_i$</span>, i.e. <span class="math-container">$\frac{\partial \mathcal{L}}{\partial w_i}$</span>, intuitively, represents the &quot;contribution&quot; of the parameter <span class="math-container">$w_i$</span> to the loss. After having computed these partial derivatives (i.e. the gradient), you use gradient descent to update each parameter <span class="math-container">$w_i$</span> as follows</p>
  <p><span class="math-container">$$
  w_i \leftarrow w_i - \gamma \frac{\partial \mathcal{L}}{\partial w_i}
  $$</span></p>
  <p>where <span class="math-container">$\frac{\partial \mathcal{L}}{\partial w_i}$</span> represents WHAT we propagatED, which is the error (or loss) that the neural network makes.</p>
  <p>This gradient descent step will hopefully make your network produce a smaller error next time.</p>
  <p>The modern version of back-propagation was published (in 1970) by a Finnish master's student called Seppo Linnainmaa, but he didn't reference neural networks. <a href="http://people.idsia.ch/%7Ejuergen/who-invented-backpropagation.html" rel="nofollow noreferrer">This Jürgen Schmidhuber's article goes into the details of the history of BP</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-10T11:47:39.903Z
Answers.Text: So you're saying the only reason it's called back propagation is because of the chain rule? I was under the impression that there is some kind of propagation of error from the output to the input, like you calculate the error on the final hidden layer, then use that to find the error in the layer before that and so on, kind of like how you propagate the input to hidden layer to output. Thanks for the answer.
Answers.UserDisplayName: Skawang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-10T12:28:41.427Z
Answers.Text: @Skawang Yes, you can view the chain rule also in that way, but that's still the chain rule, I just didn't explicitly show the "error" and how it is part of the equations, but you can find this online.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-20T13:25:54.253Z
Answers.Text: @nbro, when using chain rule for BP in deep learning, do u agree that we can start computing the gradient (i.e. the product of some firsts layer jacobians) even if the forward pass is not terminated ? I mean one can imagine an algorithm computing both forward and backward at almost the same time ?
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-20T13:51:20.910Z
Answers.Text: @RémyHosseinkhanBoucher You can compute the general **formulas** to compute the partial derivatives 
  without doing the forward pass (as I did in my answer, for example), but you need the specific values obtained during the forward pass to compute the **actual** values of the partial derivatives, otherwise, what would be the purpose of the forward pass?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-20T13:56:22.313Z
Answers.Text: So, no, you need the forward pass to compute the actual gradient. You can do this exercise with a simple neural network. Maybe a 1-hidden layer net. Compute the formulas of the backprop for it. Then you will realize that to compute the actual values you need the values obtained during the forward pass.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-06T17:04:59.470Z
Answers.LastActivityDate: 2022-04-12T08:36:03.713Z
Answers.LastEditDate: 2022-04-12T08:36:03.713Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Have a look at the following article <a href="http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html" rel="nofollow noreferrer">Principles of training multi-layer neural network using backpropagation</a>. It was very useful to me. </p>
  
  <p>You can also see <a href="https://ai.stackexchange.com/a/19945/2444">here an example of backpropagation in Matlab</a>. It effectively solves the <a href="https://en.wikipedia.org/wiki/Exclusive_or" rel="nofollow noreferrer">XOR problem</a>. You can also play around with the cost function or the learning rate. You may get surprising results! Does this answer your question?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-10T11:45:53.307Z
Answers.Text: I don't see how the method in the link you shared would work if there are activation functions on each node, it seems to give a different result to the one I referred to in the question. It makes sense if there are no activation functions though. Can you explain how the error gets propagated backward the same way the input is propagated forward? Thanks
Answers.UserDisplayName: Skawang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-10T16:43:57.303Z
Answers.Text: You need to find the derivative of the function e.g $\text{sigmoid}'(e) = s(e)*(1-s(e))$
Answers.UserDisplayName: david david
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-06T15:18:59.327Z
Answers.LastActivityDate: 2020-04-06T16:34:09.193Z
Answers.LastEditDate: 2020-04-06T16:34:09.193Z
Answers.UserDisplayName: david david
Body: <p>While looking at the mathematics of the back-propagation algorithm for a multi-layer perceptron, I noticed that in order to find the partial derivative of the cost function with respect to a weight (say <span class="math-container">$w$</span>) from any of the hidden layers, we're just writing the error function from the final outputs in terms of the inputs and hidden layer weights and then canceling all the terms without <span class="math-container">$w$</span> in it as differentiating those terms with respect to <span class="math-container">$w$</span> would give zero. </p>
  
  <p>Where is the back-propagation of error while doing this? This way, I can find the partial derivatives of the first hidden layer first and then go towards the other ones if I wanted to. Is there some other method of going about it so that the Back Propagation concept comes into play? Also, I'm looking for a general method/algorithm, not just for 1-2 hidden layers. </p>
  
  <p>I'm fairly new to this and I'm just following what's being taught in class. Nothing I found on the internet seems to have proper notation so I can't understand what they're saying.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-14T11:00:00.277Z
Comments.Text: i guess it's nothing special, just backward calculation from output loss gradient, to get loss gradients wrt every weight
Comments.UserDisplayName: Dan D.
Tags: <terminology><backpropagation><history><multilayer-perceptrons>
Title: Why is it called back-propagation?
UserDisplayName: Skawang
ThreadText: # Why is it called back-propagation?
  
  (Asked by: username_0 on 2020-04-06)
  
  While looking at the mathematics of the back-propagation algorithm for a multi-layer perceptron, I noticed that in order to find the partial derivative of the cost function with respect to a weight (say $w$) from any of the hidden layers, we're just writing the error function from the final outputs in terms of the inputs and hidden layer weights and then canceling all the terms without $w$ in it as differentiating those terms with respect to $w$ would give zero.
  
  Where is the back-propagation of error while doing this? This way, I can find the partial derivatives of the first hidden layer first and then go towards the other ones if I wanted to. Is there some other method of going about it so that the Back Propagation concept comes into play? Also, I'm looking for a general method/algorithm, not just for 1-2 hidden layers.
  
  I'm fairly new to this and I'm just following what's being taught in class. Nothing I found on the internet seems to have proper notation so I can't understand what they're saying.
  
  **Tags:** terminology, backpropagation, history, multilayer-perceptrons
  
  > **Comments:**
  > **username_1**: i guess it's nothing special, just backward calculation from output loss gradient, to get loss gradients wrt every weight  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2020-04-06. Score: 3 | Accepted answer
  
  Have a look at the following article [Principles of training multi-layer neural network using backpropagation](http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html). It was very useful to me.
  
  You can also see [here an example of backpropagation in Matlab](https://ai.stackexchange.com/a/19945/2444). It effectively solves the [XOR problem](https://en.wikipedia.org/wiki/Exclusive_or). You can also play around with the cost function or the learning rate. You may get surprising results! Does this answer your question?
  
  > **Comments:**
  > **username_0**: I don't see how the method in the link you shared would work if there are activation functions on each node, it seems to give a different result to the one I referred to in the question. It makes sense if there are no activation functions though. Can you explain how the error gets propagated backward the same way the input is propagated forward? Thanks  
  > **username_4**: You need to find the derivative of the function e.g $\text{sigmoid}'(e) = s(e)*(1-s(e))$  
  
  ---
  
  ### Answer by username_2 on 2020-04-06. Score: 8
  
  > Why is it called back-propagation?
  
  I don't think there is anything special here!
  
  It's called back-propagation (BP) because, after the forward pass, you compute the partial derivative of the loss function with respect to the parameters of the network, which, in the usual diagrams of a neural network, are placed **before** the output of the network (i.e. to the left of the output if the output of the network is on the right, or to the right if the output of the network is on the left).
  
  It's also called BP because it is just the application of the [chain rule](https://en.wikipedia.org/wiki/Chain_rule). Why is this interesting?
  
  Let me answer this question with an example. Consider the function $y=e^{\sin(x^{2})}$. This is a [composite function](https://en.wikipedia.org/wiki/Function_composition), i.e. a function composed of multiple simpler functions, which, in this case, are $e^x$, $\sin(x)$, $x^2$ and $x$. To compute the derivative of $y$ with respect to $x$, let's define the following variables
  
  \begin{align}
  y &= f(u) = e^u,\\
  u &= g(v) = \sin v = \sin(x^2),\\
  v &= h(x) = x^2
  \end{align}
  
  The derivative of $y$ with respect to the variable $x$ is (according to the [chain rule](https://en.wikipedia.org/wiki/Chain_rule))
  
  $$
  \underset{\color{red}{\LARGE \rightarrow}}{
  \frac{dy}{dx} = \frac{dy}{du} \color{green}{\cdot} \frac{du}{dv} \color{green}{\cdot}
  \frac{dv}{dx}}
  $$
  
  If you read this equation from the left to the right, you can see that we are going backward (i.e. from the function $y$ to the function $v$). This is the same thing with BP!
  
  Why is it called "chain rule"? Because you are [chaining](https://en.wikipedia.org/wiki/Chain) different partial derivatives. More specifically, you are multiplying them.
  
  BP is also known as the **reverse mode of automatic differentiation**. Why? The [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation) should be self-explanatory, given that the BP algorithm is just the computation of partial derivatives, and you do this *automatically*, i.e. with a program, rather than by hand. The expression "reverse mode" refers to the fact that we compute the derivatives from the outer function (which, in the example above, is $e^x$) to the inner function (which, in the example above, is $x$). [The Wikipedia article related to automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation) provides more details.
  
  > What exactly are you back-propagating?
  
  The partial derivative of the loss function $\mathcal{L}$ with respect to a parameter $w\_i$, i.e. $\frac{\partial \mathcal{L}}{\partial w\_i}$, intuitively, represents the "contribution" of the parameter $w\_i$ to the loss. After having computed these partial derivatives (i.e. the gradient), you use gradient descent to update each parameter $w\_i$ as follows
  
  $$
  w\_i \leftarrow w\_i - \gamma \frac{\partial \mathcal{L}}{\partial w\_i}
  $$
  
  where $\frac{\partial \mathcal{L}}{\partial w\_i}$ represents WHAT we propagatED, which is the error (or loss) that the neural network makes.
  
  This gradient descent step will hopefully make your network produce a smaller error next time.
  
  The modern version of back-propagation was published (in 1970) by a Finnish master's student called Seppo Linnainmaa, but he didn't reference neural networks. [This Jürgen Schmidhuber's article goes into the details of the history of BP](http://people.idsia.ch/%7Ejuergen/who-invented-backpropagation.html).
  
  > **Comments:**
  > **username_0**: So you're saying the only reason it's called back propagation is because of the chain rule? I was under the impression that there is some kind of propagation of error from the output to the input, like you calculate the error on the final hidden layer, then use that to find the error in the layer before that and so on, kind of like how you propagate the input to hidden layer to output. Thanks for the answer.  
  > **username_2**: @username_0 Yes, you can view the chain rule also in that way, but that's still the chain rule, I just didn't explicitly show the "error" and how it is part of the equations, but you can find this online.  
  > **username_3**: @username_2, when using chain rule for BP in deep learning, do u agree that we can start computing the gradient (i.e. the product of some firsts layer jacobians) even if the forward pass is not terminated ? I mean one can imagine an algorithm computing both forward and backward at almost the same time ?  
  > **username_2**: @username_3 You can compute the general **formulas** to compute the partial derivatives  without doing the forward pass (as I did in my answer, for example), but you need the specific values obtained during the forward pass to compute the **actual** values of the partial derivatives, otherwise, what would be the purpose of the forward pass?  
  > **username_2**: So, no, you need the forward pass to compute the actual gradient. You can do this exercise with a simple neural network. Maybe a 1-hidden layer net. Compute the formulas of the backprop for it. Then you will realize that to compute the actual values you need the values obtained during the forward pass.  
  
  ---
  
  