Answers.Body: <p>If I understand you correctly, then option one is the way multi-layer RNNs are usually implemented.</p>
  <p>A node in a multi-layer RNN <span class="math-container">$R$</span> at time <span class="math-container">$t$</span>, and layer <span class="math-container">$l$</span> (i.e. <span class="math-container">$R_{t,l}$</span>) gets 2 inputs:</p>
  <ol>
  <li><span class="math-container">$R_{t-1, l}$</span>: Same layer, but previous time-step. On the first time-step, you can take any (random or zero) input.</li>
  <li><span class="math-container">$R_{t, l-1}$</span>: Same time-step, but previous layer. On the first layer, you take the (user-)input at time t.</li>
  </ol>
  <p>This is the standard implementation. However, I don't doubt that you can do many different things, which might or might not work in certain situations. Think of bidirectional RNNs etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-23T13:41:34.870Z
Answers.Text: Thank you!! Yes - I've often thought that there is no INCORRECT way of doing it.
  
  I've tried something like this in the past - the SINGLE hidden layer - is just a mass collection of neurons - each one is connected to the input layer and provides outputs to the output layer - and then each neuron is connected to EVERY other neuron in this mass collection - picture a place of spaghetti :-).
  
  I think this didn't converge at all (or didn't seem to) and then I read that there is some benefit to formally pass values (or outputs) from one layer to the next, sequentially
Answers.UserDisplayName: Mike Arsenault
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-23T14:00:36.290Z
Answers.Text: Glad the answer helped you! If it completely solves your question, please check it as 'the correct answer' by assigning the checkmark to it. This helps future visitors of the site find what they are looking for :)
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-23T11:54:29.940Z
Answers.LastActivityDate: 2023-05-23T11:54:29.940Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I'm trying to work through a project where I'm writing my own RNN in C++ - not using any libraries. Basically I have an Input layer - 2 hidden layers - and then an output layer. In a given layer, each node collects inputs from EVERY node in the previous layer and then adding a bias, which is normal - like a normal Feed-Forward network.</p>
  <p>But here is my question - for the Recurrence, in addition to inputs from previous layer, does a node:</p>
  <ul>
  <li>ONLY get feedback from itself</li>
  <li>get feedback from every node in its own layer (this is what I'm currently doing - but it may be overkill)</li>
  <li>or can a node get feedback from nodes in layers AFTER it - I've seen architecture diagrams that suggest this.</li>
  </ul>
  <p>I'm thinking the first one is the way to go - the second one is what I'm current trying on a number of applications and very simple models just never seem to converge. I imaging the 3rd one, would be even more difficult to train.</p>
  <p>OR are all 3 correct - they are just different types of RNNs?</p>
  <p>Also, I'm assuming that nodes in the OUTPUT layer do not feedback to themselves - those nodes still do activation and have an array of weights and a bias, but they ONLY collect inputs from the last hidden layer - is that correct?</p>
  <p>Thanks
  Mike</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-22T23:35:16.557Z
Comments.Text: Finally, an interesting question!
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-23T10:56:03.003Z
Comments.Text: By "node" you mean a single unit/neuron in the recurrent layer?
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-23T13:28:25.733Z
Comments.Text: Correct - yes a single Neuron that collects inputs, multiplies by weights, adds a bias, then passes the result into an activation function
Comments.UserDisplayName: Mike Arsenault
Tags: <recurrent-neural-networks><definitions>
Title: In a Recurrent Neural Network, what are the inputs to a node in a mutli-layer RNN?
UserDisplayName: Mike Arsenault
ThreadText: # In a Recurrent Neural Network, what are the inputs to a node in a mutli-layer RNN?
  
  (Asked by: username_0 on 2023-05-22)
  
  I'm trying to work through a project where I'm writing my own RNN in C++ - not using any libraries. Basically I have an Input layer - 2 hidden layers - and then an output layer. In a given layer, each node collects inputs from EVERY node in the previous layer and then adding a bias, which is normal - like a normal Feed-Forward network.
  
  But here is my question - for the Recurrence, in addition to inputs from previous layer, does a node:
  
  * ONLY get feedback from itself
  * get feedback from every node in its own layer (this is what I'm currently doing - but it may be overkill)
  * or can a node get feedback from nodes in layers AFTER it - I've seen architecture diagrams that suggest this.
  
  I'm thinking the first one is the way to go - the second one is what I'm current trying on a number of applications and very simple models just never seem to converge. I imaging the 3rd one, would be even more difficult to train.
  
  OR are all 3 correct - they are just different types of RNNs?
  
  Also, I'm assuming that nodes in the OUTPUT layer do not feedback to themselves - those nodes still do activation and have an array of weights and a bias, but they ONLY collect inputs from the last hidden layer - is that correct?
  
  Thanks
  username_0
  
  **Tags:** recurrent-neural-networks, definitions
  
  > **Comments:**
  > **username_1**: Finally, an interesting question!  
  > **username_2**: By "node" you mean a single unit/neuron in the recurrent layer?  
  > **username_0**: Correct - yes a single Neuron that collects inputs, multiplies by weights, adds a bias, then passes the result into an activation function  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-05-23. Score: 2 | Accepted answer
  
  If I understand you correctly, then option one is the way multi-layer RNNs are usually implemented.
  
  A node in a multi-layer RNN $R$ at time $t$, and layer $l$ (i.e. $R\_{t,l}$) gets 2 inputs:
  
  1. $R\_{t-1, l}$: Same layer, but previous time-step. On the first time-step, you can take any (random or zero) input.
  2. $R\_{t, l-1}$: Same time-step, but previous layer. On the first layer, you take the (user-)input at time t.
  
  This is the standard implementation. However, I don't doubt that you can do many different things, which might or might not work in certain situations. Think of bidirectional RNNs etc.
  
  > **Comments:**
  > **username_0**: Thank you!! Yes - I've often thought that there is no INCORRECT way of doing it.  I've tried something like this in the past - the SINGLE hidden layer - is just a mass collection of neurons - each one is connected to the input layer and provides outputs to the output layer - and then each neuron is connected to EVERY other neuron in this mass collection - picture a place of spaghetti :-).  I think this didn't converge at all (or didn't seem to) and then I read that there is some benefit to formally pass values (or outputs) from one layer to the next, sequentially  
  > **username_3**: Glad the answer helped you! If it completely solves your question, please check it as 'the correct answer' by assigning the checkmark to it. This helps future visitors of the site find what they are looking for :)  
  
  ---
  
  