Explanation of Equation 5.80 in Pattern Recognition and Machine Learning - Bishop How the equation 5.80 in _Pattern Recognition and Machine Learning_ by Bishop is derived?


I tried to figure out the maths behind it and this is what I understood. The perceptron architecture having the two layers ($j$,$k$) is shown below. The activation function at node $j$ is $h(.)$ and $k$ being the output node has unit activation.

We have to find $\frac{\partial E_n}{\partial a_j^2}$. This can be evaluated as $\frac{\partial}{\partial a_j}\bigg(\frac{\partial E_n}{\partial a_j}\bigg)$. Referring the perceptron diagram and using chain rule to back propagate the error across the path from $y_k$ to $a_j$, $\frac{\partial E_n}{\partial a_j}$ can be evaluated by first differentiating $E_n$ w.r.t. $a_k$, then $a_k$ w.r.t. $z_j$ and lastly $z_j$ w.r.t. $a_j$. At node $k$, we have to consider all the incoming connections and hence have to sum over all the possible values of $k$. The expression for $\frac{\partial E_n}{\partial a_j}$ is
$$\begin{align}
\frac{\partial E_n}{\partial a_j} = \sum_{k} \frac{\partial E_n}{\partial a_k} \frac{\partial a_k}{\partial z_j} \frac{\partial z_l}{\partial a_j} = \sum_{k} \frac{\partial E_n}{\partial a_k} w_{kj} h^{'}(a_j) = h^{'}(a_j)\sum_{k} \frac{\partial E_n}{\partial a_k} w_{kj}  \tag{A}
\end{align}$$
Now, differentiating $\frac{\partial E_n}{\partial a_j}$ w.r.t. $a_j$ (using product rule of differentiation), we have
$$\begin{align}
\frac{\partial}{\partial a_j}\bigg(\frac{\partial E_n}{\partial a_j}\bigg) = \frac{\partial}{\partial a_j} \bigg(h^{'}(a_j)\sum_{k} \frac{\partial E_n}{\partial a_k} w_{kj}\bigg) = \frac{\partial}{\partial a_j} \bigg(h^{'}(a_j)\sum_{k} w_{kj}\frac{\partial E_n}{\partial a_k}\bigg)
\end{align}$$
$$\begin{align}
= h^{''}(a_j)\sum_{k} w_{kj}\frac{\partial E_n}{\partial a_k} + h^{'}(a_j)\frac{\partial}{\partial a_j} \bigg(\sum_{k} w_{kj}\frac{\partial E_n}{\partial a_k}\bigg)
\end{align}$$
$$\begin{align}
= h^{''}(a_j)\sum_{k} w_{kj}\frac{\partial E_n}{\partial a_k} + h^{'}(a_j) \sum_{k} w_{kj}\frac{\partial}{\partial a_j}\bigg(\frac{\partial E_n}{\partial a_k}\bigg) \tag{B}
\end{align}$$
The differential $\frac{\partial}{\partial a_j}\bigg(\frac{\partial E_n}{\partial a_k}\bigg)$ can be evaluated using $(A)$ as
$$\begin{align}
\frac{\partial}{\partial a_j}\bigg(\frac{\partial E_n}{\partial a_k}\bigg) = h^{'}(a_j)\sum_{k^{'}} \frac{\partial^2 E_n}{\partial a_k^{'}a_k} w_{k^{'}j}  \tag{C}
\end{align}$$
Substituting $(C)$ in $(B)$, the final expression is
$$\begin{align}
\frac{\partial^2 E_n}{\partial a_j^{2}} = h^{''}(a_j)\sum_{k} w_{kj}\frac{\partial E_n}{\partial a_k} + h^{'}(a_j)^2 \sum_{k}\sum_{k^{'}} w_{kj}w_{k^{'}j}\frac{\partial^2 E_n}{\partial a_k^{'}a_k}
\end{align}$$
