Answers.Body: <p>I will answer this question, leaving it open to challenge by anyone more knowledgeable. </p>
  
  <p>The equations to update each layer of an RNN are</p>
  
  <p><span class="math-container">$y_t = \sigma(W_x x_t + b_x )$</span></p>
  
  <p>and</p>
  
  <p><span class="math-container">$h_t = \sigma(W_hx_t + b_h)$</span></p>
  
  <p>Where <span class="math-container">$h_t$</span> is the hidden layer (in blue in the picture), and <span class="math-container">$y_t$</span> is the output layer (red in picture). This equations says that every single component in the hidden layer vector, i.e. every unit in the hidden layer, is a function of the linear combination of the <span class="math-container">$x_t$</span> vector, which is the first yellow row along the depth axis. In other words, all the first yellow input nodes on the bottom left in the picture. </p>
  
  <p>Thus, technically this picture is not correct, all the yellow nodes should have points to all the blue nodes. Also, by similar reasoning, all the blue nodes in the subsequent steps of the hidden layer should be connected to all the blue nodes of the previous layer.</p>
  
  <p>Of course, that would make for a much uglier/harder picture to make, so I don't blame the authors, although this has given me a few hours of confused research, which I guess still meets their educational goal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-07T04:49:36.697Z
Answers.LastActivityDate: 2020-04-07T04:49:36.697Z
Answers.UserDisplayName: Mike
Body: <p><a href="https://i.sstatic.net/9YIDb.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9YIDb.jpg" alt="enter image description here"></a></p>
  
  <p>This is a picture of a recurrent neural network (RNN) found on a udemy course (Deep Learning A-Z). The axis at the bottom is "time". </p>
  
  <p>In a time series problem, each yellow row from left to right would represent a sequence of a feature. In this picture, then, there are 6 sequences from 6 different features that are being fed to the network.</p>
  
  <p>I am wondering if the arrows in this picture are completely accurate in an RNN. Shouldn't every yellow node also connect to every other blue node along its depth dimension? By depth dimension here I mean the third dimensional axis of the input tensor. </p>
  
  <p>For example, the yellow node at the bottom left of this picture, which is closest to the viewer, should have an arrow pointing to all the blue nodes in the array of blue nodes that is at the very left, and not just to the blue node directly above it.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-07T05:19:15.937Z
Comments.Text: @DuttaA Thanks for the heads up, those courses definitely don't seem to be very rigorous.
Comments.UserDisplayName: Mike
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-10T19:50:20.800Z
Comments.Text: @DuttaA Yeah, I was referring to the udemy ones.
Comments.UserDisplayName: Mike
Tags: <neural-networks><recurrent-neural-networks>
Title: Is this a correct visual representation of a recurrent neural network (RNN)?
UserDisplayName: Mike
ThreadText: # Is this a correct visual representation of a recurrent neural network (RNN)?
  
  (Asked by: username_0 on 2020-04-06)
  
  [![enter image description here](https://i.sstatic.net/9YIDb.jpg)](https://i.sstatic.net/9YIDb.jpg)
  
  This is a picture of a recurrent neural network (RNN) found on a udemy course (Deep Learning A-Z). The axis at the bottom is "time".
  
  In a time series problem, each yellow row from left to right would represent a sequence of a feature. In this picture, then, there are 6 sequences from 6 different features that are being fed to the network.
  
  I am wondering if the arrows in this picture are completely accurate in an RNN. Shouldn't every yellow node also connect to every other blue node along its depth dimension? By depth dimension here I mean the third dimensional axis of the input tensor.
  
  For example, the yellow node at the bottom left of this picture, which is closest to the viewer, should have an arrow pointing to all the blue nodes in the array of blue nodes that is at the very left, and not just to the blue node directly above it.
  
  **Tags:** neural-networks, recurrent-neural-networks
  
  > **Comments:**
  > **username_0**: @DuttaA Thanks for the heads up, those courses definitely don't seem to be very rigorous.  
  > **username_0**: @DuttaA Yeah, I was referring to the udemy ones.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-04-07. Score: 0 | Accepted answer
  
  I will answer this question, leaving it open to challenge by anyone more knowledgeable.
  
  The equations to update each layer of an RNN are
  
  $y\_t = \sigma(W\_x x\_t + b\_x )$
  
  and
  
  $h\_t = \sigma(W\_hx\_t + b\_h)$
  
  Where $h\_t$ is the hidden layer (in blue in the picture), and $y\_t$ is the output layer (red in picture). This equations says that every single component in the hidden layer vector, i.e. every unit in the hidden layer, is a function of the linear combination of the $x\_t$ vector, which is the first yellow row along the depth axis. In other words, all the first yellow input nodes on the bottom left in the picture.
  
  Thus, technically this picture is not correct, all the yellow nodes should have points to all the blue nodes. Also, by similar reasoning, all the blue nodes in the subsequent steps of the hidden layer should be connected to all the blue nodes of the previous layer.
  
  Of course, that would make for a much uglier/harder picture to make, so I don't blame the authors, although this has given me a few hours of confused research, which I guess still meets their educational goal.
  
  
  ---
  
  