Answers.Body: <p>One important point I missed in first review: error is a summatory, its derivative is also a summatory.</p>
  
  <p>About offsets "b": usually they are different in each cell (if not fixed to some value, as 0). Thus, replace them by b1 and b2. Moreover, they should be optimized in the same way that the weights.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-12T18:27:50.233Z
Answers.LastActivityDate: 2018-03-12T18:48:42.753Z
Answers.LastEditDate: 2018-03-12T18:48:42.753Z
Answers.UserDisplayName: pasaba por aqui
Body: <p>I have computed the forward and backward passes of the following simple neural network, with one input, hidden, and output neurons.</p>
  <p><a href="https://i.sstatic.net/hapD0.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hapD0.jpg" alt="enter image description here" /></a></p>
  <p>Here are my computations of the forward pass.</p>
  <p><span class="math-container">\begin{align}
  net_1 &amp;= xw_{1}+b \\
  h &amp;= \sigma (net_1) \\
  net_2 &amp;= hw_{2}+b \\
  {y}' &amp;= \sigma (net_2),
  \end{align}</span></p>
  <p>where <span class="math-container">$\sigma = \frac{1}{1 + e^{-x}}$</span> (sigmoid) and <span class="math-container">$ L=\frac{1}{2}\sum(y-{y}')^{2} $</span></p>
  <p>Here are my computations of backpropagation.</p>
  <p><span class="math-container">\begin{align}
  \frac{\partial L}{\partial w_{2}}
  &amp;=\frac{\partial net_2}{\partial w_2}\frac{\partial {y}' }{\partial net_2}\frac{\partial L }{\partial {y}'} \\
  \frac{\partial L}{\partial w_{1}} &amp;= 
  \frac{\partial net_1}{\partial w_{1}} \frac{\partial h}{\partial net_1}\frac{\partial net_2}{\partial h}\frac{\partial {y}' }{\partial net_2}\frac{\partial L }{\partial {y}'}
  \end{align}</span>
  where
  <span class="math-container">\begin{align}
  \frac{\partial L }{\partial {y}'}
  &amp; =\frac{\partial (\frac{1}{2}\sum(y-{y}')^{2})}{\partial {y}'}=({y}'-y) \\
  \frac{\partial {y}' }{\partial net_2}
  &amp;={y}'(1-{y}')\\
  \frac{\partial net_2}{\partial w_2}
  &amp;= \frac{\partial(hw_{2}+b) }{\partial w_2}=h \\ 
  \frac{\partial net_2}{\partial h}
  &amp;=\frac{\partial (hw_{2}+b) }{\partial h}=w_2 \\
  \frac{\partial h}{\partial net_1}
  &amp; =h(1-h) \\
  \frac{\partial net_1}{\partial w_{1}}
  &amp;= \frac{\partial(xw_{1}+b) }{\partial w_1}=x
  \end{align}</span></p>
  <p>The gradients can be written as</p>
  <p><span class="math-container">\begin{align}
  \frac{\partial L }{\partial w_2 } &amp;=h\times {y}'(1-{y}')\times ({y}'-y) \\
  \frac{\partial L}{\partial w_{1}}
  &amp;=x\times h(1-h)\times  w_2 \times {y}'(1-{y}')\times ({y}'-y)
  \end{align}</span></p>
  <p>The weight update is</p>
  <p><span class="math-container">\begin{align}
  w_{i}^{t+1} \leftarrow w_{i}^{t}-\alpha \frac{\partial L}{\partial w_{i}}
  \end{align}</span></p>
  <p>Are my computations correct?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-12T09:09:41.907Z
Comments.Text: My review concludes that your analysis is correct. Congratulations also for the presentation. Just a minor editorial: replace "b" by b1 and b2; must be b1 and b2 also evaluated ?; clarify that sigma function is sigmoid; and better write w_2*h that h*w_2 (in this way, most of your equations are applicable to h vector and W matrix).
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-12T15:32:31.830Z
Comments.Text: @pasabaporaqui Thank you for reviewing my calcultions and yes sigma is sigmoid function. My biggest doubt is when I calculated `dnet_2/dh=w_2`. It was a surprise for me, I never thought we use weights value  to do back prop calculation. I didnt understand this part  `better write w_2*h that h*w_2 (in this way, most of your equations are applicable to h vector and W matrix)` ?
Comments.UserDisplayName: Eka
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-12T16:00:32.090Z
Comments.Text: @Eka: if the value of h is increased in 1 unit, the value of net2 is increased by w2. This is the meaning of this partial derivate.
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-12T16:02:43.560Z
Comments.Text: @Eka: do not worry about the comnent of order, it is just that is more practical/traditional write w*h than h*w. In some problems, W will be a matrix and h a vector, Wh is ok, hW is not.
Comments.UserDisplayName: pasaba por aqui
Tags: <neural-networks><backpropagation><calculus><forward-pass>
Title: Are my computations of the forward and backward pass of a neural network with one input, hidden and output neurons correct?
UserDisplayName: Eka
ThreadText: # Are my computations of the forward and backward pass of a neural network with one input, hidden and output neurons correct?
  
  (Asked by: username_0 on 2018-03-12)
  
  I have computed the forward and backward passes of the following simple neural network, with one input, hidden, and output neurons.
  
  [![enter image description here](https://i.sstatic.net/hapD0.jpg)](https://i.sstatic.net/hapD0.jpg)
  
  Here are my computations of the forward pass.
  
  \begin{align}
  net\_1 &= xw\_{1}+b \\
  h &= \sigma (net\_1) \\
  net\_2 &= hw\_{2}+b \\
  {y}' &= \sigma (net\_2),
  \end{align}
  
  where $\sigma = \frac{1}{1 + e^{-x}}$ (sigmoid) and $ L=\frac{1}{2}\sum(y-{y}')^{2} $
  
  Here are my computations of backpropagation.
  
  \begin{align}
  \frac{\partial L}{\partial w\_{2}}
  &=\frac{\partial net\_2}{\partial w\_2}\frac{\partial {y}' }{\partial net\_2}\frac{\partial L }{\partial {y}'} \\
  \frac{\partial L}{\partial w\_{1}} &=
  \frac{\partial net\_1}{\partial w\_{1}} \frac{\partial h}{\partial net\_1}\frac{\partial net\_2}{\partial h}\frac{\partial {y}' }{\partial net\_2}\frac{\partial L }{\partial {y}'}
  \end{align}
  where
  \begin{align}
  \frac{\partial L }{\partial {y}'}
  & =\frac{\partial (\frac{1}{2}\sum(y-{y}')^{2})}{\partial {y}'}=({y}'-y) \\
  \frac{\partial {y}' }{\partial net\_2}
  &={y}'(1-{y}')\\
  \frac{\partial net\_2}{\partial w\_2}
  &= \frac{\partial(hw\_{2}+b) }{\partial w\_2}=h \\
  \frac{\partial net\_2}{\partial h}
  &=\frac{\partial (hw\_{2}+b) }{\partial h}=w\_2 \\
  \frac{\partial h}{\partial net\_1}
  & =h(1-h) \\
  \frac{\partial net\_1}{\partial w\_{1}}
  &= \frac{\partial(xw\_{1}+b) }{\partial w\_1}=x
  \end{align}
  
  The gradients can be written as
  
  \begin{align}
  \frac{\partial L }{\partial w\_2 } &=h\times {y}'(1-{y}')\times ({y}'-y) \\
  \frac{\partial L}{\partial w\_{1}}
  &=x\times h(1-h)\times w\_2 \times {y}'(1-{y}')\times ({y}'-y)
  \end{align}
  
  The weight update is
  
  \begin{align}
  w\_{i}^{t+1} \leftarrow w\_{i}^{t}-\alpha \frac{\partial L}{\partial w\_{i}}
  \end{align}
  
  Are my computations correct?
  
  **Tags:** neural-networks, backpropagation, calculus, forward-pass
  > **Comments:**
  > **username_1**: My review concludes that your analysis is correct. Congratulations also for the presentation. Just a minor editorial: replace "b" by b1 and b2; must be b1 and b2 also evaluated ?; clarify that sigma function is sigmoid; and better write w_2*h that h*w_2 (in this way, most of your equations are applicable to h vector and W matrix).  
  > **username_0**: @pasabaporaqui Thank you for reviewing my calcultions and yes sigma is sigmoid function. My biggest doubt is when I calculated `dnet_2/dh=w_2`. It was a surprise for me, I never thought we use weights value  to do back prop calculation. I didnt understand this part  `better write w_2*h that h*w_2 (in this way, most of your equations are applicable to h vector and W matrix)` ?  
  > **username_1**: @Eka: if the value of h is increased in 1 unit, the value of net2 is increased by w2. This is the meaning of this partial derivate.  
  > **username_1**: @Eka: do not worry about the comnent of order, it is just that is more practical/traditional write w*h than h*w. In some problems, W will be a matrix and h a vector, Wh is ok, hW is not.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-03-12. Score: 2 | Accepted answer
  
  One important point I missed in first review: error is a summatory, its derivative is also a summatory.
  
  About offsets "b": usually they are different in each cell (if not fixed to some value, as 0). Thus, replace them by b1 and b2. Moreover, they should be optimized in the same way that the weights.
  
  
  ---
  
  