Answers.Body: <p>I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid.</p>
  
  <p>Let's say we have an Elman recurrent neural network with one input neuron, one output neuron and one hidden layer with two neurons.
  <a href="https://i.sstatic.net/XumGE.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XumGE.png" alt="Elman recurrent neural network"></a>
  In total there are 10 connections. As the image shows, <strong>neuron A receives the combined previous output of both neuron A and B as input</strong>. The same goes for neuron B. </p>
  
  <p><strong>This is not the case when we split the neurons up into multiple layers; the context neuron(s) are only used by neurons that are in the same layer.</strong> Let say we now use multiple hidden layers and keep the amount of neurons the same. In total there are 7 connections now (image below). That is 3 less than in the first example, which has only one hidden layer. So which connections do we miss? That is shown in the bottom image. (I had to paste these two images together in one image because my reputation only allows me to post 2 links)
  <a href="https://i.sstatic.net/t04vP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/t04vP.png" alt="The difference between with and without multiple hidden layers"></a></p>
  
  <p>Please note the cross; the connection between neuron A and B is not there in the first image, because it would be some kind of random recurrent connection.</p>
  
  <p><strong>The first and the last image are exactly the same. I think that if you compare the first and the last image that you agree that the statement is true.</strong></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-10T19:31:08.470Z
Answers.LastActivityDate: 2016-09-10T23:12:53.710Z
Answers.LastEditDate: 2016-09-10T23:12:53.710Z
Answers.UserDisplayName: sander2798
Body: <p>In the lecture, there was a statement:</p>
  
  <blockquote>
    <p>Recurrent neural networks with multiple hidden layers are just a special case that has some of the hidden to hidden connections missing.</p>
  </blockquote>
  
  <p>I understand recurrent means that can have connections to the previous layer and the same layer as well. </p>
  
  <p>Is there a visualization available to easily understand the above statement?</p>
  
Tags: <recurrent-neural-networks><terminology><hidden-layers>
Title: How can I understand this statement about RNNs and hidden layers?
UserDisplayName: Abhishek Bhatia
ThreadText: # How can I understand this statement about RNNs and hidden layers?
  
  (Asked by: username_0 on 2016-09-10)
  
  In the lecture, there was a statement:
  
  > Recurrent neural networks with multiple hidden layers are just a special case that has some of the hidden to hidden connections missing.
  
  I understand recurrent means that can have connections to the previous layer and the same layer as well.
  
  Is there a visualization available to easily understand the above statement?
  
  **Tags:** recurrent-neural-networks, terminology, hidden-layers
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2016-09-10. Score: 4 | Accepted answer
  
  I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid.
  
  Let's say we have an Elman recurrent neural network with one input neuron, one output neuron and one hidden layer with two neurons.
  [![Elman recurrent neural network](https://i.sstatic.net/XumGE.png)](https://i.sstatic.net/XumGE.png)
  In total there are 10 connections. As the image shows, **neuron A receives the combined previous output of both neuron A and B as input**. The same goes for neuron B.
  
  **This is not the case when we split the neurons up into multiple layers; the context neuron(s) are only used by neurons that are in the same layer.** Let say we now use multiple hidden layers and keep the amount of neurons the same. In total there are 7 connections now (image below). That is 3 less than in the first example, which has only one hidden layer. So which connections do we miss? That is shown in the bottom image. (I had to paste these two images together in one image because my reputation only allows me to post 2 links)
  [![The difference between with and without multiple hidden layers](https://i.sstatic.net/t04vP.png)](https://i.sstatic.net/t04vP.png)
  
  Please note the cross; the connection between neuron A and B is not there in the first image, because it would be some kind of random recurrent connection.
  
  **The first and the last image are exactly the same. I think that if you compare the first and the last image that you agree that the statement is true.**
  
  
  ---
  
  