Answers.Body: <p>I finally solved it out but it's long, it's not only <strong>chain rule</strong>, it includes <strong>quotient rule</strong> too. And, this is only the third last layer, once a DNN has more layers then it's more complex.</p>
  
  <p><span class="math-container">$\dfrac{dLoss}{dW_1} = \dfrac{d}{dW_1}(Out-Y) = \dfrac{d}{dW_1}Out = \dfrac{d}{dW_1}(H_4W_{13} + H_5W_{14}) \\= \dfrac{d}{dW_1}H_4W_{13} + \dfrac{d}{dW_1}H_5W_{14} \\= W_{13} \times \dfrac{d}{dW_1}H_4 + W_{14} \times \dfrac{d}{dW_1}H_5 \\= W_{13} \times \dfrac{d}{dW_1}(H_1W_7 + H_2W_8 + H_3W_9) + W_{14} \times \dfrac{d}{dW_1}(H_1W_{10} + H_2W_{11} + H_3W_{12}) \\= W_{13} \times \dfrac{d}{dW_1}H_1W_7 + W_{14} \times \dfrac{d}{dW_1}H_1W_{10} \\= W_{13}W_7 \times \dfrac{d}{dW_1}H_1 + W_{14}W_{10} \times \dfrac{d}{dW_1}H_1 \\= (W_{13}W_7 + W_{14}W_{10}) \times \dfrac{d}{dW_1}(X_1W_1 + X_2W_2) \\= (W_{13}W_7 + W_{14}W_{10}) \times X_1$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-09T08:04:22.460Z
Answers.LastActivityDate: 2019-10-09T09:26:08.010Z
Answers.LastEditDate: 2019-10-09T09:26:08.010Z
Answers.UserDisplayName: Dan D.
Body: <p>I'm trying to solve <strong>dLoss/dW1</strong>. The network is as in picture below with <strong>identity activation</strong> at all neurons:</p>
  
  <p><a href="https://i.sstatic.net/lEEKx.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lEEKx.png" alt="3-layer Network"></a></p>
  
  <p>Solving <strong>dLoss/dW7</strong> is simple as there's only 1 way to output:</p>
  
  <p><span class="math-container">$Delta = Out-Y$</span></p>
  
  <p><span class="math-container">$Loss = abs(Delta)$</span></p>
  
  <p>The case when Delta>=0, partial derivative of Loss over W7 is:</p>
  
  <p><span class="math-container">$\dfrac{dLoss}{dW_7} = \dfrac{dLoss}{dOut} \times \dfrac{dOut}{dH_4} \times \dfrac{dH_4}{dW_7} \\
  = \dfrac{d(Out-Y)}{dOut} \times \dfrac{d(H_4W_{13} + H_5W_{14})}{dH_4} \times \dfrac{d(H_1W_7 + H_2W_8 + H_3W_9)}{dW_7} \\
  = 1 \times W_{13} \times H_1$</span></p>
  
  <p>However, when solving <strong>dLoss/dW1</strong>, the situation is very different, there are 2 chains to <strong>W1</strong> through <strong>W7</strong> and <strong>W10</strong>, and now, how should the chain for <span class="math-container">$\dfrac{dLoss}{dW_1}$</span> be?</p>
  
  <p>Furthermore, at an arbitrary layer, with all outputs of all layers already calculated plus all gradients of weights on the right side also calculated, what should a formula for <span class="math-container">$\dfrac{dLoss}{dW}$</span> be?</p>
  
Tags: <math><backpropagation><gradient-descent><derivative>
Title: Backpropagation: Chain Rule to the Third Last Layer
UserDisplayName: Dan D.
ThreadText: # Backpropagation: Chain Rule to the Third Last Layer
  
  (Asked by: username_0 on 2019-10-09)
  
  I'm trying to solve **dLoss/dW1**. The network is as in picture below with **identity activation** at all neurons:
  
  [![3-layer Network](https://i.sstatic.net/lEEKx.png)](https://i.sstatic.net/lEEKx.png)
  
  Solving **dLoss/dW7** is simple as there's only 1 way to output:
  
  $Delta = Out-Y$
  
  $Loss = abs(Delta)$
  
  The case when Delta>=0, partial derivative of Loss over W7 is:
  
  $\dfrac{dLoss}{dW\_7} = \dfrac{dLoss}{dOut} \times \dfrac{dOut}{dH\_4} \times \dfrac{dH\_4}{dW\_7} \\
  = \dfrac{d(Out-Y)}{dOut} \times \dfrac{d(H\_4W\_{13} + H\_5W\_{14})}{dH\_4} \times \dfrac{d(H\_1W\_7 + H\_2W\_8 + H\_3W\_9)}{dW\_7} \\
  = 1 \times W\_{13} \times H\_1$
  
  However, when solving **dLoss/dW1**, the situation is very different, there are 2 chains to **W1** through **W7** and **W10**, and now, how should the chain for $\dfrac{dLoss}{dW\_1}$ be?
  
  Furthermore, at an arbitrary layer, with all outputs of all layers already calculated plus all gradients of weights on the right side also calculated, what should a formula for $\dfrac{dLoss}{dW}$ be?
  
  **Tags:** math, backpropagation, gradient-descent, derivative
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-10-09. Score: 0 | Accepted answer
  
  I finally solved it out but it's long, it's not only **chain rule**, it includes **quotient rule** too. And, this is only the third last layer, once a DNN has more layers then it's more complex.
  
  $\dfrac{dLoss}{dW\_1} = \dfrac{d}{dW\_1}(Out-Y) = \dfrac{d}{dW\_1}Out = \dfrac{d}{dW\_1}(H\_4W\_{13} + H\_5W\_{14}) \\= \dfrac{d}{dW\_1}H\_4W\_{13} + \dfrac{d}{dW\_1}H\_5W\_{14} \\= W\_{13} \times \dfrac{d}{dW\_1}H\_4 + W\_{14} \times \dfrac{d}{dW\_1}H\_5 \\= W\_{13} \times \dfrac{d}{dW\_1}(H\_1W\_7 + H\_2W\_8 + H\_3W\_9) + W\_{14} \times \dfrac{d}{dW\_1}(H\_1W\_{10} + H\_2W\_{11} + H\_3W\_{12}) \\= W\_{13} \times \dfrac{d}{dW\_1}H\_1W\_7 + W\_{14} \times \dfrac{d}{dW\_1}H\_1W\_{10} \\= W\_{13}W\_7 \times \dfrac{d}{dW\_1}H\_1 + W\_{14}W\_{10} \times \dfrac{d}{dW\_1}H\_1 \\= (W\_{13}W\_7 + W\_{14}W\_{10}) \times \dfrac{d}{dW\_1}(X\_1W\_1 + X\_2W\_2) \\= (W\_{13}W\_7 + W\_{14}W\_{10}) \times X\_1$
  
  
  ---
  
  