Answers.Body: <p>For the first problem your <span class="math-container">$O_2$</span> is mistaken and should be corrected as below
  <span class="math-container">$$
  \begin{aligned}
  O_2 &amp;= \text{Sig} \left( -0.6 - 0.1 \cdot 2 + 0.2 \cdot 6 \right) \\
  &amp;= \text{Sig} \left( 0.4 \right) \\
  &amp;\approx 0.5987 \\
  \end{aligned}
  $$</span></p>
  <p><span class="math-container">$$
  \begin{aligned}
  O_3 &amp;= \text{Sig} \left( 0.6 - 0.3 \cdot 0.5987 + 1.6 \cdot 0.5 \right) \\
  &amp;\approx 0.7721 \\
  \end{aligned}
  $$</span></p>
  <p><span class="math-container">$$
  \begin{aligned}
  p\left(C=1 | x; w,b\right) = O_3 \approx 0.7721 
  \end{aligned}
  $$</span></p>
  <p>For the second problem thus corrected to:
  <span class="math-container">$$
  \begin{aligned}
  p\left(C=0 | x; w,b\right) = 1 - \left(C=1 | x; w,b\right) = 0.2279
  \end{aligned}
  $$</span></p>
  <p>For the third problem since <span class="math-container">$w_6$</span> is one of the weights connecting the output layer and we assume <span class="math-container">$\hat{y}=g(z^{(3)})$</span>, where <span class="math-container">$z^{(3)}$</span> is the net input for the output unit and <span class="math-container">$g$</span> is its sigmoid transfer function. Then we arrive at:
  <span class="math-container">$$
  \begin{aligned}
  \frac{\delta L}{\delta w_6} &amp;= \frac{\delta L}{\delta \hat{y}} \cdot \frac{\delta \hat{y}}{\delta z^{(3)}} \cdot \frac{\delta z^{(3)}}{\delta w_6} \\
  \end{aligned}
  $$</span></p>
  <p>and since <span class="math-container">$L$</span> is the binary cross-entropy loss function, we know <span class="math-container">$\frac{\delta L}{\delta \hat{y}} = \frac{(\hat{y}-y)}{(1-\hat{y})\hat{y}}$</span>. And from above transfer function we know <span class="math-container">$\frac{\delta \hat{y}}{\delta z^{(3)}}=(1-\hat{y})\hat{y}$</span>, and <span class="math-container">$\frac{\delta z^{(3)}}{\delta w_6}=O_2$</span>. Thus finally we simplify as <span class="math-container">$\frac{\delta L}{\delta w_6}=(O_3-y)O_2$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-05T15:30:12.937Z
Answers.Text: Thanks for your response. One more thing, for $\frac{\delta L}{\delta w_4}$ is it $\frac{\delta L}{\delta \hat{y}} \cdot \frac{\delta \hat{y}}{\delta z^{(3)}} \cdot \frac{\delta z^{(3)}}{\delta z^{(2)}} \cdot \frac{ \delta z^{(2)}}{\delta w_4} = (O_3-y)(1-O_2)O_2x_2$?
Answers.UserDisplayName: Airlangga Fidiyanto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-06T00:55:42.947Z
Answers.Text: For the hidden layer it's more complex and you need to further chain $\frac{\delta z^{(3)}}{\delta z^{(2)}}$ into $\frac{\delta z^{(3)}}{\delta O_2} \cdot \frac{\delta O_2}{\delta z^{(2)}}$, and then try to simplify from there.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-05T02:37:53.477Z
Answers.LastActivityDate: 2022-12-05T03:11:53.490Z
Answers.LastEditDate: 2022-12-05T03:11:53.490Z
Answers.UserDisplayName: cinch
Body: <p>Given a neural network model for Covid-19 classification with <span class="math-container">$C=1$</span> for positive and <span class="math-container">$C=0$</span> for negative
  <a href="https://i.sstatic.net/aGz1r.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/aGz1r.png" alt="enter image description here" /></a>
  Let <span class="math-container">$x_1 = 6$</span> and <span class="math-container">$x_2=2$</span> find</p>
  <ol>
  <li>Probability if the patient got Covid-19 <span class="math-container">$p\left(C=1 | x; w,b\right)$</span></li>
  <li>Probability if the patient didn’t get Covid-19 <span class="math-container">$p\left(C=0 | x; w,b\right)$</span></li>
  <li>Find the gradient of <span class="math-container">$CE_{Loss}$</span></li>
  </ol>
  <p>My attempt</p>
  <p><strong>For the first problem</strong>
  <span class="math-container">$$
  \begin{aligned}
  O_1 &amp;= \text{ReLU} \left( b_1 + \sum{x_iw_i} \right) \\
  &amp;= \text{ReLU} \left( 0.2 + 6 \cdot 0.3 - 0.2 \cdot 2 \right) \\
  &amp;= \text{ReLU} \left( 1.6 \right) \\
  &amp;= 1.6 \\
  \end{aligned}
  $$</span></p>
  <p><span class="math-container">$$
  \begin{aligned}
  O_2 &amp;= \text{Sig} \left( -0.6 - 0.1 \cdot 2 - 0.2 \cdot 6 \right) \\
  &amp;= \text{Sig} \left( -2 \right) \\
  &amp;\approx 0.8808 \\
  \end{aligned}
  $$</span></p>
  <p><span class="math-container">$$
  \begin{aligned}
  O_3 &amp;= \text{Sig} \left( 0.6 - 0.3 \cdot 0.8808 + 1.6 \cdot 0.5 \right) \\
  &amp;\approx 0.75690 \\
  \end{aligned}
  $$</span></p>
  <p><span class="math-container">$$
  \begin{aligned}
  p\left(C=1 | x; w,b\right) = O_3 \approx 0.75690
  \end{aligned}
  $$</span></p>
  <p><strong>For the second problem</strong>
  <span class="math-container">$$
  \begin{aligned}
  p\left(C=0 | x; w,b\right) = 1 - \left(C=1 | x; w,b\right) = 0.2431
  \end{aligned}
  $$</span></p>
  <p><strong>For the third problem</strong></p>
  <p>Since it is a lot of things to calculate I'll take <span class="math-container">$\frac{\delta L}{\delta w_6}$</span> as example
  <span class="math-container">$$
  \begin{aligned}
  \frac{\delta L}{\delta w_6} &amp;= \frac{\delta L}{\delta \hat{y}} \cdot \frac{\delta \hat{y}}{\delta w_6} \\
  &amp;= \left(\hat{y}-y\right)O_2\left(O_3 \left(1-O_3\right)\right)\\
  &amp;= \left(O_3-y\right)O_2\left(O_3 \left(1-O_3\right)\right)
  \end{aligned}
  $$</span>
  The answer key says it should be <span class="math-container">$\left(O_3-y\right)O_2$</span>. Where did I go wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-04T17:27:04.653Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Tags: <neural-networks><homework>
Title: My cross entropy loss gradient calculation is wrong according to the answer key
UserDisplayName: Airlangga Fidiyanto
ThreadText: # My cross entropy loss gradient calculation is wrong according to the answer key
  
  (Asked by: username_0 on 2022-12-04)
  
  Given a neural network model for Covid-19 classification with $C=1$ for positive and $C=0$ for negative
  [![enter image description here](https://i.sstatic.net/aGz1r.png)](https://i.sstatic.net/aGz1r.png)
  Let $x\_1 = 6$ and $x\_2=2$ find
  
  1. Probability if the patient got Covid-19 $p\left(C=1 | x; w,b\right)$
  2. Probability if the patient didn’t get Covid-19 $p\left(C=0 | x; w,b\right)$
  3. Find the gradient of $CE\_{Loss}$
  
  My attempt
  
  **For the first problem**
  $$
  \begin{aligned}
  O\_1 &= \text{ReLU} \left( b\_1 + \sum{x\_iw\_i} \right) \\
  &= \text{ReLU} \left( 0.2 + 6 \cdot 0.3 - 0.2 \cdot 2 \right) \\
  &= \text{ReLU} \left( 1.6 \right) \\
  &= 1.6 \\
  \end{aligned}
  $$
  
  $$
  \begin{aligned}
  O\_2 &= \text{Sig} \left( -0.6 - 0.1 \cdot 2 - 0.2 \cdot 6 \right) \\
  &= \text{Sig} \left( -2 \right) \\
  &\approx 0.8808 \\
  \end{aligned}
  $$
  
  $$
  \begin{aligned}
  O\_3 &= \text{Sig} \left( 0.6 - 0.3 \cdot 0.8808 + 1.6 \cdot 0.5 \right) \\
  &\approx 0.75690 \\
  \end{aligned}
  $$
  
  $$
  \begin{aligned}
  p\left(C=1 | x; w,b\right) = O\_3 \approx 0.75690
  \end{aligned}
  $$
  
  **For the second problem**
  $$
  \begin{aligned}
  p\left(C=0 | x; w,b\right) = 1 - \left(C=1 | x; w,b\right) = 0.2431
  \end{aligned}
  $$
  
  **For the third problem**
  
  Since it is a lot of things to calculate I'll take $\frac{\delta L}{\delta w\_6}$ as example
  $$
  \begin{aligned}
  \frac{\delta L}{\delta w\_6} &= \frac{\delta L}{\delta \hat{y}} \cdot \frac{\delta \hat{y}}{\delta w\_6} \\
  &= \left(\hat{y}-y\right)O\_2\left(O\_3 \left(1-O\_3\right)\right)\\
  &= \left(O\_3-y\right)O\_2\left(O\_3 \left(1-O\_3\right)\right)
  \end{aligned}
  $$
  The answer key says it should be $\left(O\_3-y\right)O\_2$. Where did I go wrong?
  
  **Tags:** neural-networks, homework
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-12-05. Score: 3 | Accepted answer
  
  For the first problem your $O\_2$ is mistaken and should be corrected as below
  $$
  \begin{aligned}
  O\_2 &= \text{Sig} \left( -0.6 - 0.1 \cdot 2 + 0.2 \cdot 6 \right) \\
  &= \text{Sig} \left( 0.4 \right) \\
  &\approx 0.5987 \\
  \end{aligned}
  $$
  
  $$
  \begin{aligned}
  O\_3 &= \text{Sig} \left( 0.6 - 0.3 \cdot 0.5987 + 1.6 \cdot 0.5 \right) \\
  &\approx 0.7721 \\
  \end{aligned}
  $$
  
  $$
  \begin{aligned}
  p\left(C=1 | x; w,b\right) = O\_3 \approx 0.7721
  \end{aligned}
  $$
  
  For the second problem thus corrected to:
  $$
  \begin{aligned}
  p\left(C=0 | x; w,b\right) = 1 - \left(C=1 | x; w,b\right) = 0.2279
  \end{aligned}
  $$
  
  For the third problem since $w\_6$ is one of the weights connecting the output layer and we assume $\hat{y}=g(z^{(3)})$, where $z^{(3)}$ is the net input for the output unit and $g$ is its sigmoid transfer function. Then we arrive at:
  $$
  \begin{aligned}
  \frac{\delta L}{\delta w\_6} &= \frac{\delta L}{\delta \hat{y}} \cdot \frac{\delta \hat{y}}{\delta z^{(3)}} \cdot \frac{\delta z^{(3)}}{\delta w\_6} \\
  \end{aligned}
  $$
  
  and since $L$ is the binary cross-entropy loss function, we know $\frac{\delta L}{\delta \hat{y}} = \frac{(\hat{y}-y)}{(1-\hat{y})\hat{y}}$. And from above transfer function we know $\frac{\delta \hat{y}}{\delta z^{(3)}}=(1-\hat{y})\hat{y}$, and $\frac{\delta z^{(3)}}{\delta w\_6}=O\_2$. Thus finally we simplify as $\frac{\delta L}{\delta w\_6}=(O\_3-y)O\_2$
  
  > **Comments:**
  > **username_0**: Thanks for your response. One more thing, for $\frac{\delta L}{\delta w_4}$ is it $\frac{\delta L}{\delta \hat{y}} \cdot \frac{\delta \hat{y}}{\delta z^{(3)}} \cdot \frac{\delta z^{(3)}}{\delta z^{(2)}} \cdot \frac{ \delta z^{(2)}}{\delta w_4} = (O_3-y)(1-O_2)O_2x_2$?  
  > **username_2**: For the hidden layer it's more complex and you need to further chain $\frac{\delta z^{(3)}}{\delta z^{(2)}}$ into $\frac{\delta z^{(3)}}{\delta O_2} \cdot \frac{\delta O_2}{\delta z^{(2)}}$, and then try to simplify from there.  
  
  ---
  
  