Answers.Body: <p>Yes, a neural network plus loss function can be viewed as a function composition as you have written, and back propagation is just the chain function repeated. Your equations <span class="math-container">$L = f(g(h(\dots u(v(\dots))))$</span> and <span class="math-container">$\frac{\partial L}{\partial v} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\dots\frac{\partial u}{\partial v}$</span> are useful for high level intuition.</p>
  
  <p>At some point you need to look at the actual forms of the functions, and that introduces a bit more complexity. For instance, the loss function for a mini-batch can be expressed in terms of the output. This is the equivalent of <span class="math-container">$L = f(g)$</span> for a batch or mini-batch with mean squared error loss, where I am using <span class="math-container">$J$</span> (for cost) in place of <span class="math-container">$L$</span> and the output of the neural network <span class="math-container">$\hat{y}$</span> in place of <span class="math-container">$g$</span>:</p>
  
  <p><span class="math-container">$$J = \frac{1}{2N}\sum_{i=1}^{N}(\hat{y}_i - y_i)^2$$</span></p>
  
  <p>The gradient of <span class="math-container">$J$</span> with respect to <span class="math-container">$\hat{y}$</span> is equivalent to your first part <span class="math-container">$\frac{\partial f}{\partial g}$</span>:</p>
  
  <p><span class="math-container">$$\nabla_{\hat{y}} J = \frac{1}{N}\sum_{i=1}^{N}(\hat{y}_i - y_i)$$</span></p>
  
  <p>Many of the functions in a neural network involve sums over terms. They can be expressed as vector and matrix operations, which can make them look simpler, but you still need to have code somewhere that works through all the elements.</p>
  
  <p>There is one thing that the function composition view does not show well. The gradient you want to calculate is <span class="math-container">$\nabla_{\theta} J$</span>, where <span class="math-container">$\theta$</span> represents all the <em>parameters</em> of the neural network (weights and biases). The parameters in each layer are end points of back propagation - they are not functions of anything else, and the chain rule has to stop with them. That means you have a series of "dead ends" - or possibly another way of thinking about it would be that your <span class="math-container">$\frac{\partial L}{\partial v} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\dots\frac{\partial u}{\partial v}$</span> is the "trunk" of the algorithm that links layers together, and every couple of steps there is a "branch" to calculate the gradient of the parameters that you want to change.</p>
  
  <p>More concretely, if you have weight parameters in each layer noted as <span class="math-container">$W^{(n)}$</span>, and two functions for each layer (the sum over weights times inputs, and an activation function) then your example ends up looking like this progression:</p>
  
  <p><span class="math-container">$$\frac{\partial L}{\partial W^{(n)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial W^{(n)}}$$</span></p>
  
  <p><span class="math-container">$$\frac{\partial L}{\partial W^{(n-1)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial i}\frac{\partial i}{\partial j}  \frac{\partial j}{\partial W^{(n-1)}}$$</span></p>
  
  <p><span class="math-container">$$\frac{\partial L}{\partial W^{(n-2)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial i}\frac{\partial i}{\partial j}\frac{\partial j}{\partial k}\frac{\partial k}{\partial l}  \frac{\partial l}{\partial W^{(n-2)}}$$</span></p>
  
  <p>. . . this is the same idea but showing the goal of calculating <span class="math-container">$\nabla_{W} L$</span> which doesn't fit into a <em>single</em> chain of gradients. Notice the last term on each line does not appear in the next line. However, you can keep all the terms prior to that and re-use them in the next line - this matches the layer-by-layer calculations in many implementations of back propagation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-27T08:11:07.660Z
Answers.LastActivityDate: 2018-11-27T08:47:18.337Z
Answers.LastEditDate: 2018-11-27T08:47:18.337Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm wondering if I can visualize the backprop process as follows (please excuse me if I have written something terrible wrong). If the loss function <span class="math-container">$L$</span> on a neural network represents the function has the form 
  <span class="math-container">$$L = f(g(h(\dots u(v(\dots))))$$</span>
  then we can visualize the derivative of <span class="math-container">$L$</span> wrt the <span class="math-container">$i$</span>th function <span class="math-container">$v$</span> as 
  <span class="math-container">$$\frac{\partial L}{\partial v} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\dots\frac{\partial u}{\partial v}.$$</span></p>
  
  <p>Am I able to view all neural networks as having a loss function of the form of <span class="math-container">$L$</span> given above? That is, am I correct in saying that any neural network is just a function composition and that I can write the partial derivative wrt any parameter as written above (I know I took the partial with respect to the function <span class="math-container">$v$</span>). </p>
  
  <p>Thanks</p>
  
Tags: <neural-networks><backpropagation>
Title: Am I able to visualize the differentiation in backprop as follows?
UserDisplayName: sadlyfe
ThreadText: # Am I able to visualize the differentiation in backprop as follows?
  
  (Asked by: username_0 on 2018-11-26)
  
  I'm wondering if I can visualize the backprop process as follows (please excuse me if I have written something terrible wrong). If the loss function $L$ on a neural network represents the function has the form
  $$L = f(g(h(\dots u(v(\dots))))$$
  then we can visualize the derivative of $L$ wrt the $i$th function $v$ as
  $$\frac{\partial L}{\partial v} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\dots\frac{\partial u}{\partial v}.$$
  
  Am I able to view all neural networks as having a loss function of the form of $L$ given above? That is, am I correct in saying that any neural network is just a function composition and that I can write the partial derivative wrt any parameter as written above (I know I took the partial with respect to the function $v$).
  
  Thanks
  
  **Tags:** neural-networks, backpropagation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-11-27. Score: 2 | Accepted answer
  
  Yes, a neural network plus loss function can be viewed as a function composition as you have written, and back propagation is just the chain function repeated. Your equations $L = f(g(h(\dots u(v(\dots))))$ and $\frac{\partial L}{\partial v} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\dots\frac{\partial u}{\partial v}$ are useful for high level intuition.
  
  At some point you need to look at the actual forms of the functions, and that introduces a bit more complexity. For instance, the loss function for a mini-batch can be expressed in terms of the output. This is the equivalent of $L = f(g)$ for a batch or mini-batch with mean squared error loss, where I am using $J$ (for cost) in place of $L$ and the output of the neural network $\hat{y}$ in place of $g$:
  
  $$J = \frac{1}{2N}\sum\_{i=1}^{N}(\hat{y}\_i - y\_i)^2$$
  
  The gradient of $J$ with respect to $\hat{y}$ is equivalent to your first part $\frac{\partial f}{\partial g}$:
  
  $$\nabla\_{\hat{y}} J = \frac{1}{N}\sum\_{i=1}^{N}(\hat{y}\_i - y\_i)$$
  
  Many of the functions in a neural network involve sums over terms. They can be expressed as vector and matrix operations, which can make them look simpler, but you still need to have code somewhere that works through all the elements.
  
  There is one thing that the function composition view does not show well. The gradient you want to calculate is $\nabla\_{\theta} J$, where $\theta$ represents all the *parameters* of the neural network (weights and biases). The parameters in each layer are end points of back propagation - they are not functions of anything else, and the chain rule has to stop with them. That means you have a series of "dead ends" - or possibly another way of thinking about it would be that your $\frac{\partial L}{\partial v} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\dots\frac{\partial u}{\partial v}$ is the "trunk" of the algorithm that links layers together, and every couple of steps there is a "branch" to calculate the gradient of the parameters that you want to change.
  
  More concretely, if you have weight parameters in each layer noted as $W^{(n)}$, and two functions for each layer (the sum over weights times inputs, and an activation function) then your example ends up looking like this progression:
  
  $$\frac{\partial L}{\partial W^{(n)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial W^{(n)}}$$
  
  $$\frac{\partial L}{\partial W^{(n-1)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial i}\frac{\partial i}{\partial j} \frac{\partial j}{\partial W^{(n-1)}}$$
  
  $$\frac{\partial L}{\partial W^{(n-2)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial i}\frac{\partial i}{\partial j}\frac{\partial j}{\partial k}\frac{\partial k}{\partial l} \frac{\partial l}{\partial W^{(n-2)}}$$
  
  . . . this is the same idea but showing the goal of calculating $\nabla\_{W} L$ which doesn't fit into a *single* chain of gradients. Notice the last term on each line does not appear in the next line. However, you can keep all the terms prior to that and re-use them in the next line - this matches the layer-by-layer calculations in many implementations of back propagation.
  
  
  ---
  
  