Body: <p>I am writing my own recurrent neural network in Java to understand the inner workings better. While working through the math, I found that in timesteps later than 2 the gradient of weight w of neuron n depends on the gradients of all neurons at all timesteps before. A handwritten example is given, I tried to write as clearly as possible.</p>
  
  <p>Could anyone verify this so I can finish my network? Am I missing a piece or is my premise wrong, as in the output of a neuron is s(Wx + Vh + b), where h is the last step of <em>only</em> neuron n?</p>
  
  <p><a href="https://i.sstatic.net/QFMfz.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QFMfz.jpg" alt="The feedforward formulas for the first three timesteps"></a>
  <a href="https://i.sstatic.net/jW9mq.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jW9mq.jpg" alt="My derivations of the gradients of the specific weight 1 of neuron 1, just as an example"></a></p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-05T08:05:06.290Z
Comments.Text: are you sure that is not your homework? why are you doing it in java python makes life easier.
Comments.UserDisplayName: Steve Okay
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-05T13:29:39.503Z
Comments.Text: Yes, I'm sure that this is not my homework, in fact I don't have any course related to this subject :). And firstly, Java is my preferred language, I _can_ use other languages but I like Java the most. And speed i no concern for me. Secondly, I want to use this in another application which has to be in Java, so its just easier. Python may make dealing with arrays and numpy dealing with tensors easier, but I already got fairly good workarounds for that.
Comments.UserDisplayName: JustAGuy
Tags: <recurrent-neural-networks><gradient-descent>
Title: Are gradients of weights in RNNs dependent on the gradient of every neuron in that layer?
UserDisplayName: JustAGuy
ThreadText: # Are gradients of weights in RNNs dependent on the gradient of every neuron in that layer?
  
  (Asked by: username_0 on 2017-08-04)
  
  I am writing my own recurrent neural network in Java to understand the inner workings better. While working through the math, I found that in timesteps later than 2 the gradient of weight w of neuron n depends on the gradients of all neurons at all timesteps before. A handwritten example is given, I tried to write as clearly as possible.
  
  Could anyone verify this so I can finish my network? Am I missing a piece or is my premise wrong, as in the output of a neuron is s(Wx + Vh + b), where h is the last step of *only* neuron n?
  
  [![The feedforward formulas for the first three timesteps](https://i.sstatic.net/QFMfz.jpg)](https://i.sstatic.net/QFMfz.jpg)
  [![My derivations of the gradients of the specific weight 1 of neuron 1, just as an example](https://i.sstatic.net/jW9mq.jpg)](https://i.sstatic.net/jW9mq.jpg)
  
  **Tags:** recurrent-neural-networks, gradient-descent
  
  > **Comments:**
  > **username_1**: are you sure that is not your homework? why are you doing it in java python makes life easier.  
  > **username_0**: Yes, I'm sure that this is not my homework, in fact I don't have any course related to this subject :). And firstly, Java is my preferred language, I _can_ use other languages but I like Java the most. And speed i no concern for me. Secondly, I want to use this in another application which has to be in Java, so its just easier. Python may make dealing with arrays and numpy dealing with tensors easier, but I already got fairly good workarounds for that.  
  
  ---
  
  ## 0 answers
  
  