Answers.Body: <p>The first image is correct. The information will flow from left to right in each layer and from top to bottom in between layers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T10:59:21.713Z
Answers.Text: Thanks for the comment I take it that there should be 5 nodes in the picture, not 4? Also good point on the hidden state - I think I'm missing this part as well - is the hidden state passed to the dense layer and the y value (prediction)?
Answers.UserDisplayName: Mindaugas Bernatavičius
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T11:42:25.700Z
Answers.Text: @SajanGohil yes you are right, I haven't noticed the return sequence=True on the second layer. I will update the answer.
Answers.UserDisplayName: razvanc92
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T11:47:59.817Z
Answers.Text: @MindaugasBernatavičius Yes there should had been 5 nodes instead of 4. I have removed the picture since it's not mandatory anymore. Yes the hidden state is passed to the Dense layer. If you select return_sequences=True, each hidden state from the last RNN layer is passed through the dense layer. If you set return_sequences to false, then only the last hidden state from the last RNN layer is sent to the Dense layer.
Answers.UserDisplayName: razvanc92
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T10:41:08.433Z
Answers.LastActivityDate: 2020-12-08T11:45:04.387Z
Answers.LastEditDate: 2020-12-08T11:45:04.387Z
Answers.UserDisplayName: razvanc92
Body: <p>I'm trying to create a simple blogpost on RNNs, that should give a better insight into how they work in Keras. Let's say:</p>
  <pre><code>model = keras.models.Sequential()
  model.add(keras.layers.SimpleRNN(5, return_sequences=True, input_shape=[None, 1]))
  model.add(keras.layers.SimpleRNN(5, return_sequences=True))
  model.add(keras.layers.Dense(1))
  </code></pre>
  <p>I came up with the following visualization (this is only a sketch), which I'm quite unsure about:</p>
  <p><a href="https://i.sstatic.net/3sIX7.png" rel="noreferrer"><img src="https://i.sstatic.net/3sIX7.png" alt="enter image description here" /></a></p>
  <p>The RNN architecture is comprised of 3 layers represented in the picture.</p>
  <p><strong>Question:</strong> is this correct? Is the input &quot;flowing&quot; thought each layer neuron to neuron or only though the layers, like in the picture below. Is there anything else that is not correct - any other visualizations to look into?</p>
  <p><a href="https://i.sstatic.net/QaM1A.png" rel="noreferrer"><img src="https://i.sstatic.net/QaM1A.png" alt="enter image description here" /></a></p>
  <p><strong>Update:</strong> my assumptions are based on my understanding from what I saw in Geron's book. The recurrent neurons are connected, see: <a href="https://pasteboard.co/JDXTFVw.png" rel="noreferrer">https://pasteboard.co/JDXTFVw.png</a> ... he then proceeds to talk about connections between different layers, see: <a href="https://pasteboard.co/JDXTXcz.png" rel="noreferrer">https://pasteboard.co/JDXTXcz.png</a> - did I misunderstand him or is it just a peculiarity in keras framework?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-08T11:49:43.873Z
Comments.Text: As you can see in fig 15-2, neurons in a layer don't communicate with each other, it is unrolled version in the 2nd part of 15-2, ie. it is the same layer across each timestep, there is no next layer in that diagram. In fig 15-1, a single neuron is shown (unrolled on right). What's happening for a single neuron in 15-1 is happening to every neuron in 15-2, there is no concept of multiple layers introduced in the images you referenced, they are different timesteps for the same neuron/layer.
Comments.UserDisplayName: SajanGohil
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-08T12:45:11.977Z
Comments.Text: Yea, I was getting the 2nd part of the argument you presented, now I also understand the first one - the confusion creeped in because the example was using a single neuron RNN as a simplified self-contained network and then added several cells in a single layer RNN. As you said - both pictures show single layer RNN unrolled through time, the difference being the first one has a single neuron and the other has multiple neurons in that layer. Thanks
Comments.UserDisplayName: Mindaugas Bernatavičius
Tags: <keras><recurrent-neural-networks><data-visualization>
Title: How to graphically represent a RNN architecture implemented in Keras?
UserDisplayName: Mindaugas Bernatavičius
ThreadText: # How to graphically represent a RNN architecture implemented in Keras?
  
  (Asked by: username_0 on 2020-12-08)
  
  I'm trying to create a simple blogpost on RNNs, that should give a better insight into how they work in Keras. Let's say:
  model = keras.models.Sequential()
  model.add(keras.layers.SimpleRNN(5, return_sequences=True, input_shape=[None, 1]))
  model.add(keras.layers.SimpleRNN(5, return_sequences=True))
  model.add(keras.layers.Dense(1))
  
  I came up with the following visualization (this is only a sketch), which I'm quite unsure about:
  
  The RNN architecture is comprised of 3 layers represented in the picture.
  Question: is this correct? Is the input "flowing" thought each layer neuron to neuron or only though the layers, like in the picture below. Is there anything else that is not correct - any other visualizations to look into?
  
  Update: my assumptions are based on my understanding from what I saw in Geron's book. The recurrent neurons are connected, see: https://pasteboard.co/JDXTFVw.png ... he then proceeds to talk about connections between different layers, see: https://pasteboard.co/JDXTXcz.png - did I misunderstand him or is it just a peculiarity in keras framework?
  
  **Tags:** keras, recurrent-neural-networks, data-visualization
  > **Comments:**
  > **username_1**: As you can see in fig 15-2, neurons in a layer don't communicate with each other, it is unrolled version in the 2nd part of 15-2, ie. it is the same layer across each timestep, there is no next layer in that diagram. In fig 15-1, a single neuron is shown (unrolled on right). What's happening for a single neuron in 15-1 is happening to every neuron in 15-2, there is no concept of multiple layers introduced in the images you referenced, they are different timesteps for the same neuron/layer.  
  > **username_0**: Yea, I was getting the 2nd part of the argument you presented, now I also understand the first one - the confusion creeped in because the example was using a single neuron RNN as a simplified self-contained network and then added several cells in a single layer RNN. As you said - both pictures show single layer RNN unrolled through time, the difference being the first one has a single neuron and the other has multiple neurons in that layer. Thanks  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-12-08. Score: 0
  
  The first image is correct. The information will flow from left to right in each layer and from top to bottom in between layers.
  
  > **Comments:**
  > **username_0**: Thanks for the comment I take it that there should be 5 nodes in the picture, not 4? Also good point on the hidden state - I think I'm missing this part as well - is the hidden state passed to the dense layer and the y value (prediction)?  
  > **username_2**: @SajanGohil yes you are right, I haven't noticed the return sequence=True on the second layer. I will update the answer.  
  > **username_2**: @MindaugasBernatavičius Yes there should had been 5 nodes instead of 4. I have removed the picture since it's not mandatory anymore. Yes the hidden state is passed to the Dense layer. If you select return_sequences=True, each hidden state from the last RNN layer is passed through the dense layer. If you set return_sequences to false, then only the last hidden state from the last RNN layer is sent to the Dense layer.  
  
  ---
  
  