Answers.Body: <p>The author is rather free with changing from row to column format. The main philosophy or framework seems to be to implement the directions of &quot;forward&quot; evaluation and &quot;backwards&quot; gradient differentiation in the left-right direction, in diagrams as well as in formulas.</p>
  <p>However, this philosophy is broken several times, for instance in writing <span class="math-container">$a=f(z)$</span> instead of <span class="math-container">$f(z)=a$</span>, or in using weight matrices that are indexed for matrix-vector multiplication, that is, the usual right-to-left direction <span class="math-container">$(z^{(3)})^T=\theta^{(2)} (a^{(2)})^T$</span>, which following the philosophy should be written as <span class="math-container">$a^{(2)}(\theta^{(2)})^T=z^{(3)}$</span>.</p>
  <p>But then again the philosophy gets reversed in formulas like
  <span class="math-container">$$
  \delta^{(l)} = \delta^{(l + 1)}\,\Theta^{(l)}\,f'\left( a^{(l)} \right)
  $$</span>
  which clearly is right-to-left, which suggests that the gradients <span class="math-container">$δ^{(l)}$</span> are row vectors and the argument vectors like <span class="math-container">$a^{(l)}$</span> are column vectors.</p>
  <p>In short, it's a mess.</p>
  <hr />
  <p>Despite that, your questions have direct answers without relying too much on what directional philosophy is used</p>
  <p>Q1. <span class="math-container">$f'(a^{(3)})$</span> as used and positioned relative to other vectors and matrices is the diagonal matrix with the entries <span class="math-container">$f'(a_j^{(3)})$</span> on the diagonal. This comes out as component-wise product in matrix-vector or vector-matrix multiplications.</p>
  <p>Q2. If you were to discuss that, you would need another index in all the formulas indicating the training sample. Such as <span class="math-container">$J(x^{[k]},\theta)$</span> as the residual of the net output to <span class="math-container">$y^{[k]}$</span>. The gradient of the sum <span class="math-container">$\frac1m\sum_{k=1}^mJ(x^{[k]},\theta)$</span> would be the sum of all single gradients that get computed independently of each other. Another interpretation of the factor <span class="math-container">$\frac1m$</span> is that it is the gradient at the top level <span class="math-container">$J$</span> that gets propagated backwards to the gradients of each variable.</p>
  <p>Q3. Of course some doubt is appropriate. As the activated value is a function of the linear combined input, <span class="math-container">$a^{(3)}=f(z^{(3)})$</span>, so is the derivative <span class="math-container">$\frac{\partial a^{(3)}}{\partial z^{(3)}}=f'(z^{(3)})$</span>. Both must have the same arguments. This is just a typo, perhaps copy-pasted some times.</p>
  <hr />
  <p>I've tried how to consistently implement the left-to-right philosophy, but it is too cumbersome. One would have to use something unfamiliar like some kind of reverse polish notation, so instead of <span class="math-container">$v=\phi(u,w)$</span> one would have to write <span class="math-container">$[u,w:\,\phi]=v$</span> or similar. So it is better to stay with right-to-left consistently, as also the author ended up doing. Thus <span class="math-container">$x,z,a$</span> are column vectors, gradients (against the tradition in differential calculus) are row vectors. In algorithmic differentiation it is one tradition to denote tangent vectors for forward differentiation with a dot, <span class="math-container">$\dot x, \dot z,\dot a$</span>, and gradients that get pushed back with a bar, so <span class="math-container">$\bar x,\bar z=\delta, \bar a$</span>.</p>
  <p>The construction principle for gradient propagation is that if <span class="math-container">$v=\phi(u,w)$</span>, then the relation of tangents pushed forward to the level of before and after the operation <span class="math-container">$\phi$</span> and gradients pushed back to that stage satisfy
  <span class="math-container">$$
  \bar v\dot v=\bar u\dot u+\bar w\dot w.
  $$</span>
  Inserting <span class="math-container">$\dot v=\phi_{,u}\dot u+\phi_{,w}\dot w$</span> results in
  <span class="math-container">$$
  \bar v\phi_{,u}\dot u+\bar v\phi_{,w}\dot w=\bar u\dot u+\bar w\dot w.
  $$</span>
  Comparing both sides gives <span class="math-container">$\bar u=\bar v\phi_{,u}$</span>, <span class="math-container">$\bar w=\bar v\phi_{,w}$</span>.</p>
  <p>In a wider context <span class="math-container">$\bar w$</span> is a linear functional, meaning with scalar value, of <span class="math-container">$\dot w$</span>. So if <span class="math-container">$w$</span> is a matrix, then the linear functional is obtained via the trace, <span class="math-container">${\rm Tr}(\bar w·\dot w)$</span>. So if for instance <span class="math-container">$\phi(u,w)=w·u$</span> in a matrix-vector product, then by the product rule <span class="math-container">$\dot v=w·\dot u+\dot w·u$</span> and
  <span class="math-container">$$
  {\rm Tr}(\bar w·\dot w)=\bar v·\dot w·u={\rm Tr}(u·\bar v·\dot w),
  $$</span>
  so the comparison gives <span class="math-container">$\bar w=u·\bar v$</span>.</p>
  <hr />
  <p>The example network in atomic formulas is
  <span class="math-container">\begin{align}
  z^{(2)}&amp;=\Theta^{(1)}·x       \\
  a^{(2)}&amp;=f(z^{(2)})           \\
  z^{(3)}&amp;=\Theta^{(2)}·a^{(2)} \\
  a^{(3)}&amp;=f(z^{(3)})           \\
  \end{align}</span>
  and then <span class="math-container">$J$</span> is computed via some loss function from <span class="math-container">$a^{(3)}$</span> and the reference value <span class="math-container">$y$</span>.</p>
  <p>Starting from the gradient <span class="math-container">$\bar a^{(3)}$</span> computed from the loss function, the pushed-back gradients compute as
  <span class="math-container">\begin{align}
  \bar z^{(3)} &amp;= \bar a^{(3)}·{\rm diag}(f'(z^{(3)}))  \\
  \bar a^{(2)} &amp;= \bar z^{(3)}·\Theta^{(2)}             \\
  \bar \Theta^{(2)} &amp;= a^{(2)}·\bar z^{(3)}             \\
  \bar z^{(2)} &amp;= \bar a^{(2)}·{\rm diag}(f'(z^{(2)}))  \\
  \bar x &amp;= \bar z^{(2)}·\Theta^{(1)}             \\
  \bar \Theta^{(1)} &amp;= x·\bar z^{(2)}             \\
  \end{align}</span></p>
  <p>Of course one can combine some of these formulas, like
  <span class="math-container">$$
  \delta^{(2)}=\bar z^{(2)}
  =\bar z^{(3)}·\Theta^{(2)}·{\rm diag}(f'(z^{(2)}))
  =\delta^{(3)}·\Theta^{(2)}·{\rm diag}(f'(z^{(2)}))
  $$</span>
  and if <span class="math-container">$J=\frac12\sum |a_j^{(3)}-y_j|^2$</span>, then also
  <span class="math-container">$$
  \delta^{(3)}=\bar z^{(3)}=\bar J\,[a_1^{(3)}-y_1,a_2^{(3)}-y_2,…]·{\rm diag}(f'(z^{(3)}))
  $$</span>
  with for instance <span class="math-container">$\bar J=\frac1m$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-23T10:19:40.417Z
Answers.LastActivityDate: 2022-11-23T11:12:20.170Z
Answers.LastEditDate: 2022-11-23T11:12:20.170Z
Answers.UserDisplayName: Lutz Lehmann
Answers.Body: <p>Re your Q1 &amp; Q3: assuming single training example for now, indeed you're right mathematically speaking <span class="math-container">$f^{'}(a^{(3)})$</span> shouldn't be a constant scalar and from the author's derivation section above your referenced equation (3), this derivative of activation function in the same layer should be evaluated at different values of <span class="math-container">$z_1^{(3)}, z_2^{(3)}$</span> as the (weighted sum) net input for the 2 demonstrated neurons at the last output layer (and they've been already computed in previous feedforward pass), thus confirms your Q3's 2nd interpretation that the derivative of the activation function is evaluated with the values of the weighted sum <span class="math-container">$z_i$</span> that is actually passed through the activation function. In fact you may also refer to <a href="https://en.wikipedia.org/wiki/Delta_rule" rel="nofollow noreferrer">Delta rule</a> which is a special case of backpropagation algorithm which confirms the same interpretation.</p>
  <blockquote>
  <p>For a neuron <span class="math-container">$j$</span> with activation function <span class="math-container">${g(x)}$</span>, the delta rule for neuron <span class="math-container">$j$</span>'s <span class="math-container">$i$</span>th weight <span class="math-container">$w_{ji}$</span> is given by <span class="math-container">$\Delta w_{ji}=\alpha (t_{j}-y_{j})g'(h_{j})x_{i}$</span>, where <span class="math-container">$\alpha$</span>  is a small constant called learning rate, <span class="math-container">$g(x)$</span> is the neuron's activation function, <span class="math-container">$g'$</span> is the derivative of <span class="math-container">$g$</span>, <span class="math-container">$t_{j}$</span> is the target output, <span class="math-container">$h_{j}$</span> is the weighted sum of the neuron's inputs, <span class="math-container">$y_{j}$</span> is the actual output, <span class="math-container">$x_{i}$</span> is the <span class="math-container">$i$</span>th input.</p>
  </blockquote>
  <p>Finally <span class="math-container">$\delta^{(3)}=[\delta_1^{(3)}, \space \delta_2^{(3)}]$</span> is a <span class="math-container">$1 \times 2$</span> vector representing &quot;error&quot; terms for the same 2 output neurons which can be confirmed by the author's conclusion section in more abstract linear algebra notations. Thus it's best to view <span class="math-container">$f^{'}(a^{(3)})$</span> as a <span class="math-container">$2 \times 2$</span> square matrix with eigenvalues identical to <span class="math-container">$f^{'}(z_1^{(3)})$</span> and <span class="math-container">$f^{'}(z_2^{(3)})$</span>, respectively, and we should actually treat the explicit vector in your equation (3) as a <span class="math-container">$ 1 \times 2$</span> vector and then the final result matches as <span class="math-container">$[y_1-a_1^{(3)}, \space y_2-a_2^{(3)}]f^{'}(a^{(3)}) = \delta^{(3)}$</span>.</p>
  <p>As for your final Q2 when training multiple examples using scaled loss function you don't usually need to calculate any additional average, the scaled loss function already takes care to minimize the mean square error. The only difference now is you're now dealing with much larger vectors/matrices such as your equation (3). Say you have 3 training sets, then the above <span class="math-container">$\delta^{(3)}$</span> will be a <span class="math-container">$ 1 \times 6$</span> vector and <span class="math-container">$f^{'}(a^{(3)})$</span> will be a <span class="math-container">$ 6 \times 6$</span> matrix. Basically you start with the same random small values for all the same 8 weights in above same network architecture, but you'll have to compute larger vectors and matrices during both feedforward and back propagations of any epoch.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-23T04:28:56.690Z
Answers.LastActivityDate: 2022-11-23T04:51:49.357Z
Answers.LastEditDate: 2022-11-23T04:51:49.357Z
Answers.UserDisplayName: cinch
Body: <p>I am reading about backpropagation for fully connected neural networks and I found a very interesting <a href="https://www.jeremyjordan.me/neural-networks-training/" rel="nofollow noreferrer">article</a> by Jeremy Jordan. It explains the process from start to finish. There is a section though that confused me a bit. The partial derivative of the cost function (MSE) with regard to the <span class="math-container">$\theta_{jk}^{(2)}$</span> weights is:</p>
  <p><span class="math-container">$$\frac{\partial J(\theta)}{\partial \theta_{jk}^{(2)}} =  \left( \frac{\partial J(\theta)}{\partial a_j^{(3)}}\right) \left( \frac{\partial a_j^{(3)}}{\partial z_j^{(3)}}\right) \left(\frac{\partial z_j^{(3)}}{\partial \theta_{jk}^{(2)}} \right) \tag{1}$$</span></p>
  <p>The article defines the next equation as the &quot;error&quot; term. The equation <span class="math-container">$eq:2$</span> is the combination of the first two partials in the chain rule:</p>
  <p><span class="math-container">$$ \delta_i^{(3)}  = \frac {1}{m} (y_i - a_i^{(3)}) f^{'}(a^{(3)}) \tag{2}$$</span></p>
  <p>Where:</p>
  <ul>
  <li><span class="math-container">$ i: $</span> The index of the neuron in the layer</li>
  <li><span class="math-container">$ ^{(3)}: $</span> Denotes the layer (in this case 3 is the output layer)</li>
  <li><span class="math-container">$ z_i: $</span> The weighted sum of the inputs of the <span class="math-container">$i_{th}$</span> neuron</li>
  <li><span class="math-container">$ m: $</span> The number of training samples</li>
  <li><span class="math-container">$ y_i: $</span> The expected value of the <span class="math-container">$ i_{th} $</span> neuron</li>
  <li><span class="math-container">$ a_i: $</span> The predicted value of the <span class="math-container">$ i_{th} $</span> neuron</li>
  <li><span class="math-container">$ f^{'}: $</span> The derivative of the activation function</li>
  </ul>
  <p>So a few lines after the definition above the article states:</p>
  <blockquote>
  <p><span class="math-container">$ \delta^{(3)} $</span> is a vector of length j where j is equal to the number of output neurons
  <span class="math-container">$$ \delta^{(3)} =
   \begin{bmatrix}
   y_1 - a_1^{(3)} \newline
   y_2 - a_2^{(3)} \newline
   \cdots \newline
   y_j - a_j^{(3)} \newline
   \end{bmatrix} f^{'}(a^{(3)})  \tag{3}
  $$</span></p>
  </blockquote>
  <p>Q1. I strongly suspect that the  <span class="math-container">$ f^{'}(a^{(3)}) $</span> is a vector of length <span class="math-container">$j$</span> and not a scalar. Basically, it is a vector containing the derivative of the activation function for every neuron of the output layer. How is it possible in <span class="math-container">$eq:3$</span> to multiply it with another vector and still get a vector and not a <span class="math-container">$j\ x\ j$</span> matrix? Is the multiplication elementwise?</p>
  <p>Q2. How is the <span class="math-container">$ f^{'}(a^{(3)}) $</span> calculated for every neuron for multiple training samples? From what I understand, while training with batches I would have to average the <span class="math-container">$ (y_i - a_i^{(3)}) $</span> term for the whole batch for every neuron. So in fact the term <span class="math-container">$ (y_i - a_i^{(3)}) $</span> is the sum for the whole batch and that's why the <span class="math-container">$ \frac {1}{m} $</span> is present.  Does that apply to the derivative too? Meaning do I have to calculate the average of the derivative for the whole batch for each neuron?</p>
  <p>Q3. What does <span class="math-container">$ f^{'}(a^{(3)}) $</span> actually mean? Is this the derivative of the activation function evaluated with the values of the <span class="math-container">$a_i^{(3)}$</span> outputs? Or is it the derivative of the activation function evaluated with the values of the weighted sum <span class="math-container">$ z_i $</span> that is actually passed through the activation function to produce the <span class="math-container">$a_i^{(3)} = f(z_i)$</span> output? And if the second would I have to keep track of the average of the <span class="math-container">$z_i$</span> for each neuron in order to obtain the average of the <span class="math-container">$ f^{'} $</span></p>
  
Tags: <backpropagation>
Title: Back propagation activation function derivative
UserDisplayName: Kyriafinis Vasilis
ThreadText: # Back propagation activation function derivative
  
  (Asked by: username_0 on 2022-11-21)
  
  I am reading about backpropagation for fully connected neural networks and I found a very interesting [article](https://www.jeremyjordan.me/neural-networks-training/) by Jeremy Jordan. It explains the process from start to finish. There is a section though that confused me a bit. The partial derivative of the cost function (MSE) with regard to the $\theta\_{jk}^{(2)}$ weights is:
  
  $$\frac{\partial J(\theta)}{\partial \theta\_{jk}^{(2)}} = \left( \frac{\partial J(\theta)}{\partial a\_j^{(3)}}\right) \left( \frac{\partial a\_j^{(3)}}{\partial z\_j^{(3)}}\right) \left(\frac{\partial z\_j^{(3)}}{\partial \theta\_{jk}^{(2)}} \right) \tag{1}$$
  
  The article defines the next equation as the "error" term. The equation $eq:2$ is the combination of the first two partials in the chain rule:
  
  $$ \delta\_i^{(3)} = \frac {1}{m} (y\_i - a\_i^{(3)}) f^{'}(a^{(3)}) \tag{2}$$
  
  Where:
  
  * $ i: $ The index of the neuron in the layer
  * $ ^{(3)}: $ Denotes the layer (in this case 3 is the output layer)
  * $ z\_i: $ The weighted sum of the inputs of the $i\_{th}$ neuron
  * $ m: $ The number of training samples
  * $ y\_i: $ The expected value of the $ i\_{th} $ neuron
  * $ a\_i: $ The predicted value of the $ i\_{th} $ neuron
  * $ f^{'}: $ The derivative of the activation function
  
  So a few lines after the definition above the article states:
  
  > $ \delta^{(3)} $ is a vector of length j where j is equal to the number of output neurons
  > $$ \delta^{(3)} =
  > \begin{bmatrix}
  > y\_1 - a\_1^{(3)} \newline
  > y\_2 - a\_2^{(3)} \newline
  > \cdots \newline
  > y\_j - a\_j^{(3)} \newline
  > \end{bmatrix} f^{'}(a^{(3)}) \tag{3}
  > $$
  
  Q1. I strongly suspect that the $ f^{'}(a^{(3)}) $ is a vector of length $j$ and not a scalar. Basically, it is a vector containing the derivative of the activation function for every neuron of the output layer. How is it possible in $eq:3$ to multiply it with another vector and still get a vector and not a $j\ x\ j$ matrix? Is the multiplication elementwise?
  
  Q2. How is the $ f^{'}(a^{(3)}) $ calculated for every neuron for multiple training samples? From what I understand, while training with batches I would have to average the $ (y\_i - a\_i^{(3)}) $ term for the whole batch for every neuron. So in fact the term $ (y\_i - a\_i^{(3)}) $ is the sum for the whole batch and that's why the $ \frac {1}{m} $ is present. Does that apply to the derivative too? Meaning do I have to calculate the average of the derivative for the whole batch for each neuron?
  
  Q3. What does $ f^{'}(a^{(3)}) $ actually mean? Is this the derivative of the activation function evaluated with the values of the $a\_i^{(3)}$ outputs? Or is it the derivative of the activation function evaluated with the values of the weighted sum $ z\_i $ that is actually passed through the activation function to produce the $a\_i^{(3)} = f(z\_i)$ output? And if the second would I have to keep track of the average of the $z\_i$ for each neuron in order to obtain the average of the $ f^{'} $
  
  **Tags:** backpropagation
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-11-23. Score: 0 | Accepted answer
  
  The author is rather free with changing from row to column format. The main philosophy or framework seems to be to implement the directions of "forward" evaluation and "backwards" gradient differentiation in the left-right direction, in diagrams as well as in formulas.
  
  However, this philosophy is broken several times, for instance in writing $a=f(z)$ instead of $f(z)=a$, or in using weight matrices that are indexed for matrix-vector multiplication, that is, the usual right-to-left direction $(z^{(3)})^T=\theta^{(2)} (a^{(2)})^T$, which following the philosophy should be written as $a^{(2)}(\theta^{(2)})^T=z^{(3)}$.
  
  But then again the philosophy gets reversed in formulas like
  $$
  \delta^{(l)} = \delta^{(l + 1)}\,\Theta^{(l)}\,f'\left( a^{(l)} \right)
  $$
  which clearly is right-to-left, which suggests that the gradients $δ^{(l)}$ are row vectors and the argument vectors like $a^{(l)}$ are column vectors.
  
  In short, it's a mess.
  
  ---
  
  Despite that, your questions have direct answers without relying too much on what directional philosophy is used
  
  Q1. $f'(a^{(3)})$ as used and positioned relative to other vectors and matrices is the diagonal matrix with the entries $f'(a\_j^{(3)})$ on the diagonal. This comes out as component-wise product in matrix-vector or vector-matrix multiplications.
  
  Q2. If you were to discuss that, you would need another index in all the formulas indicating the training sample. Such as $J(x^{[k]},\theta)$ as the residual of the net output to $y^{[k]}$. The gradient of the sum $\frac1m\sum\_{k=1}^mJ(x^{[k]},\theta)$ would be the sum of all single gradients that get computed independently of each other. Another interpretation of the factor $\frac1m$ is that it is the gradient at the top level $J$ that gets propagated backwards to the gradients of each variable.
  
  Q3. Of course some doubt is appropriate. As the activated value is a function of the linear combined input, $a^{(3)}=f(z^{(3)})$, so is the derivative $\frac{\partial a^{(3)}}{\partial z^{(3)}}=f'(z^{(3)})$. Both must have the same arguments. This is just a typo, perhaps copy-pasted some times.
  
  ---
  
  I've tried how to consistently implement the left-to-right philosophy, but it is too cumbersome. One would have to use something unfamiliar like some kind of reverse polish notation, so instead of $v=\phi(u,w)$ one would have to write $[u,w:\,\phi]=v$ or similar. So it is better to stay with right-to-left consistently, as also the author ended up doing. Thus $x,z,a$ are column vectors, gradients (against the tradition in differential calculus) are row vectors. In algorithmic differentiation it is one tradition to denote tangent vectors for forward differentiation with a dot, $\dot x, \dot z,\dot a$, and gradients that get pushed back with a bar, so $\bar x,\bar z=\delta, \bar a$.
  
  The construction principle for gradient propagation is that if $v=\phi(u,w)$, then the relation of tangents pushed forward to the level of before and after the operation $\phi$ and gradients pushed back to that stage satisfy
  $$
  \bar v\dot v=\bar u\dot u+\bar w\dot w.
  $$
  Inserting $\dot v=\phi\_{,u}\dot u+\phi\_{,w}\dot w$ results in
  $$
  \bar v\phi\_{,u}\dot u+\bar v\phi\_{,w}\dot w=\bar u\dot u+\bar w\dot w.
  $$
  Comparing both sides gives $\bar u=\bar v\phi\_{,u}$, $\bar w=\bar v\phi\_{,w}$.
  
  In a wider context $\bar w$ is a linear functional, meaning with scalar value, of $\dot w$. So if $w$ is a matrix, then the linear functional is obtained via the trace, ${\rm Tr}(\bar w·\dot w)$. So if for instance $\phi(u,w)=w·u$ in a matrix-vector product, then by the product rule $\dot v=w·\dot u+\dot w·u$ and
  $$
  {\rm Tr}(\bar w·\dot w)=\bar v·\dot w·u={\rm Tr}(u·\bar v·\dot w),
  $$
  so the comparison gives $\bar w=u·\bar v$.
  
  ---
  
  The example network in atomic formulas is
  \begin{align}
  z^{(2)}&=\Theta^{(1)}·x \\
  a^{(2)}&=f(z^{(2)}) \\
  z^{(3)}&=\Theta^{(2)}·a^{(2)} \\
  a^{(3)}&=f(z^{(3)}) \\
  \end{align}
  and then $J$ is computed via some loss function from $a^{(3)}$ and the reference value $y$.
  
  Starting from the gradient $\bar a^{(3)}$ computed from the loss function, the pushed-back gradients compute as
  \begin{align}
  \bar z^{(3)} &= \bar a^{(3)}·{\rm diag}(f'(z^{(3)})) \\
  \bar a^{(2)} &= \bar z^{(3)}·\Theta^{(2)} \\
  \bar \Theta^{(2)} &= a^{(2)}·\bar z^{(3)} \\
  \bar z^{(2)} &= \bar a^{(2)}·{\rm diag}(f'(z^{(2)})) \\
  \bar x &= \bar z^{(2)}·\Theta^{(1)} \\
  \bar \Theta^{(1)} &= x·\bar z^{(2)} \\
  \end{align}
  
  Of course one can combine some of these formulas, like
  $$
  \delta^{(2)}=\bar z^{(2)}
  =\bar z^{(3)}·\Theta^{(2)}·{\rm diag}(f'(z^{(2)}))
  =\delta^{(3)}·\Theta^{(2)}·{\rm diag}(f'(z^{(2)}))
  $$
  and if $J=\frac12\sum |a\_j^{(3)}-y\_j|^2$, then also
  $$
  \delta^{(3)}=\bar z^{(3)}=\bar J\,[a\_1^{(3)}-y\_1,a\_2^{(3)}-y\_2,…]·{\rm diag}(f'(z^{(3)}))
  $$
  with for instance $\bar J=\frac1m$.
  
  
  ---
  
  ### Answer by username_2 on 2022-11-23. Score: 0
  
  Re your Q1 & Q3: assuming single training example for now, indeed you're right mathematically speaking $f^{'}(a^{(3)})$ shouldn't be a constant scalar and from the author's derivation section above your referenced equation (3), this derivative of activation function in the same layer should be evaluated at different values of $z\_1^{(3)}, z\_2^{(3)}$ as the (weighted sum) net input for the 2 demonstrated neurons at the last output layer (and they've been already computed in previous feedforward pass), thus confirms your Q3's 2nd interpretation that the derivative of the activation function is evaluated with the values of the weighted sum $z\_i$ that is actually passed through the activation function. In fact you may also refer to [Delta rule](https://en.wikipedia.org/wiki/Delta_rule) which is a special case of backpropagation algorithm which confirms the same interpretation.
  
  > For a neuron $j$ with activation function ${g(x)}$, the delta rule for neuron $j$'s $i$th weight $w\_{ji}$ is given by $\Delta w\_{ji}=\alpha (t\_{j}-y\_{j})g'(h\_{j})x\_{i}$, where $\alpha$ is a small constant called learning rate, $g(x)$ is the neuron's activation function, $g'$ is the derivative of $g$, $t\_{j}$ is the target output, $h\_{j}$ is the weighted sum of the neuron's inputs, $y\_{j}$ is the actual output, $x\_{i}$ is the $i$th input.
  
  Finally $\delta^{(3)}=[\delta\_1^{(3)}, \space \delta\_2^{(3)}]$ is a $1 \times 2$ vector representing "error" terms for the same 2 output neurons which can be confirmed by the author's conclusion section in more abstract linear algebra notations. Thus it's best to view $f^{'}(a^{(3)})$ as a $2 \times 2$ square matrix with eigenvalues identical to $f^{'}(z\_1^{(3)})$ and $f^{'}(z\_2^{(3)})$, respectively, and we should actually treat the explicit vector in your equation (3) as a $ 1 \times 2$ vector and then the final result matches as $[y\_1-a\_1^{(3)}, \space y\_2-a\_2^{(3)}]f^{'}(a^{(3)}) = \delta^{(3)}$.
  
  As for your final Q2 when training multiple examples using scaled loss function you don't usually need to calculate any additional average, the scaled loss function already takes care to minimize the mean square error. The only difference now is you're now dealing with much larger vectors/matrices such as your equation (3). Say you have 3 training sets, then the above $\delta^{(3)}$ will be a $ 1 \times 6$ vector and $f^{'}(a^{(3)})$ will be a $ 6 \times 6$ matrix. Basically you start with the same random small values for all the same 8 weights in above same network architecture, but you'll have to compute larger vectors and matrices during both feedforward and back propagations of any epoch.
  
  
  ---
  
  