Answers.Body: <p>According to this <a href="https://arxiv.org/pdf/1303.5778.pdf" rel="nofollow noreferrer">paper</a> - which discusses RNNs, LSTMs, Stacked RNNs, and even Bidirectional RNNs - a one-layer RNN is built as follows (eq 1 and 2):
  <span class="math-container">$$\begin{align}
  h_t &amp;= \mathcal H(W_{xh} x_t + W_{hh}h_{t-1}+b_h)\\
  y_t &amp;= W_{hy}h_t + b_y
  \end{align}$$</span>
  These two lines should be iterated for the sequence length <span class="math-container">$T$</span>, so you get as output hidden states <span class="math-container">$h=(h_1,...,h_T)$</span> and sequences <span class="math-container">$y=(y_1,...,y_T)$</span>. <span class="math-container">$\mathcal{H}$</span> is the activation function.</p>
  <p>According to the paper:</p>
  <blockquote>
  <p>Deep RNNs can be created by stacking multiple RNN hidden layers on top of each other, with the output sequence of one layer forming the input sequence for the
  next.</p>
  </blockquote>
  <p>This means that the recurrence is at the level of computing the hidden state, <span class="math-container">$h_t$</span>, meaning that if you have <span class="math-container">$N$</span> recurrent layers you compute all the <span class="math-container">$h^n$</span> sequences iteratively, and then when at the end you compute the output sequence <span class="math-container">$y$</span> which is also the network's output. See Eq. 11 and 12.</p>
  <p>In general, I've seen that RNNs have different implementations. For example, in Keras the recurrence occurs on the output sequence <span class="math-container">$y$</span>: so you compute all the output sequences, and optionally also return the states (if so these will inizialize the states of the next layer), which are fed to the next recurrent layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T17:33:46.867Z
Answers.LastActivityDate: 2023-05-26T17:33:46.867Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am confused by where the recurrence happens in RNNs, especially in the context of deep neural networks. I am trying to transform an ordinary neural network into a recurrent one from scratch.</p>
  <pre><code>nn = NeuralNetwork()
  nn.add(Layer(2, 2)) # fully connected layer
  nn.add(ReLU()) # activation layer/function
  nn.add(Layer(2, 1))
  nn.add(Tanh())
  </code></pre>
  <p><a href="https://i.sstatic.net/duLZk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/duLZk.png" alt="rnn" /></a></p>
  <p>Does the recurrence happen at the last hidden layer of the network or after each fully connected layer?</p>
  <p>I have heard of <a href="https://d2l.ai/chapter_recurrent-modern/deep-rnn.html" rel="nofollow noreferrer">stacking</a> RNNs, but I am not sure why we can't simply add a layer to an existing RNN. Is it because the recurrence is after the activation function/layer, hence the following code won't work?</p>
  <pre><code>import numpy as np
  
  
  class Layer():
    '''
      Fullly connected layer
    '''
  
    def __init__(self, input_size, output_size, randomize=True):
      # initialize weights and biases
      if randomize:
        self.W = np.random.randn(input_size, output_size)
      else:
        self.W = np.ones((input_size, output_size))
      self.b = np.zeros((1, output_size))
  
    def forward(self, X):
      return np.dot(X, self.W) + self.b
  
  
  class NeuralNetwork():
    '''
      Neural network divided into layers
    '''
  
    def __init__(self):
      self.layers = []
  
    def add(self, layer):
      '''
        Add layer
      '''
      self.layers.append(layer)
  
    def forward(self, X):
      '''
        Propagate input forward through each layer
      '''
      for layer in self.layers:
        # output of current layer becomes input of next layer
        X = layer.forward(X)
      return X
  
  class ReLU():
    '''
      ReLU activation layer
    '''
  
    def forward(self, X):
      self.X = X
      return np.maximum(0, X)
  
  
  class RecurrentLayer():
  
    def __init__(self, input_size, output_size, randomize=True):
      self.layer = Layer(input_size, output_size, randomize)
      if randomize:
        self.V = np.random.randn(output_size, output_size)
      else:
        self.V = np.ones((output_size, output_size))
    
    def forward(self, X_seq):
      hidden_state = []
      for x_t in X_seq:
        h_t = self.layer.forward(x_t)
        try:
          h_t += np.dot(hidden_state[-1], self.V)
        except:
          pass
        hidden_state.append(h_t)
      return hidden_state
  
  X_sequence = [[1, 1], [2, 2], [3, 3]]
  nn = NeuralNetwork()
  nn.add(RecurrentLayer(2, 2, randomize=False))
  nn.add(ReLU())
  nn.add(RecurrentLayer(2, 1, randomize=False))
  
  nn.forward(X_sequence)
  </code></pre>
  <pre><code>[array([[4.]]), array([[20.]]), array([[64.]])]
  </code></pre>
  
Tags: <recurrent-neural-networks>
Title: RNNs - Is the recurrence at the layer-level or at the network-level?
UserDisplayName: BPDev
ThreadText: # RNNs - Is the recurrence at the layer-level or at the network-level?
  
  (Asked by: username_0 on 2023-05-26)
  
  I am confused by where the recurrence happens in RNNs, especially in the context of deep neural networks. I am trying to transform an ordinary neural network into a recurrent one from scratch.
  
  ```
  nn = NeuralNetwork()
  nn.add(Layer(2, 2)) # fully connected layer
  nn.add(ReLU()) # activation layer/function
  nn.add(Layer(2, 1))
  nn.add(Tanh())
  
  ```
  
  [![rnn](https://i.sstatic.net/duLZk.png)](https://i.sstatic.net/duLZk.png)
  
  Does the recurrence happen at the last hidden layer of the network or after each fully connected layer?
  
  I have heard of [stacking](https://d2l.ai/chapter_recurrent-modern/deep-rnn.html) RNNs, but I am not sure why we can't simply add a layer to an existing RNN. Is it because the recurrence is after the activation function/layer, hence the following code won't work?
  
  ```
  import numpy as np
  
  class Layer():
    '''
      Fullly connected layer
    '''
  
    def __init__(self, input_size, output_size, randomize=True):
      # initialize weights and biases
      if randomize:
        self.W = np.random.randn(input_size, output_size)
      else:
        self.W = np.ones((input_size, output_size))
      self.b = np.zeros((1, output_size))
  
    def forward(self, X):
      return np.dot(X, self.W) + self.b
  
  class NeuralNetwork():
    '''
      Neural network divided into layers
    '''
  
    def __init__(self):
      self.layers = []
  
    def add(self, layer):
      '''
        Add layer
      '''
      self.layers.append(layer)
  
    def forward(self, X):
      '''
        Propagate input forward through each layer
      '''
      for layer in self.layers:
        # output of current layer becomes input of next layer
        X = layer.forward(X)
      return X
  
  class ReLU():
    '''
      ReLU activation layer
    '''
  
    def forward(self, X):
      self.X = X
      return np.maximum(0, X)
  
  class RecurrentLayer():
  
    def __init__(self, input_size, output_size, randomize=True):
      self.layer = Layer(input_size, output_size, randomize)
      if randomize:
        self.V = np.random.randn(output_size, output_size)
      else:
        self.V = np.ones((output_size, output_size))
    
    def forward(self, X_seq):
      hidden_state = []
      for x_t in X_seq:
        h_t = self.layer.forward(x_t)
        try:
          h_t += np.dot(hidden_state[-1], self.V)
        except:
          pass
        hidden_state.append(h_t)
      return hidden_state
  
  X_sequence = [[1, 1], [2, 2], [3, 3]]
  nn = NeuralNetwork()
  nn.add(RecurrentLayer(2, 2, randomize=False))
  nn.add(ReLU())
  nn.add(RecurrentLayer(2, 1, randomize=False))
  
  nn.forward(X_sequence)
  
  ```
  
  ```
  [array([[4.]]), array([[20.]]), array([[64.]])]
  
  ```
  
  **Tags:** recurrent-neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-26. Score: 0 | Accepted answer
  
  According to this [paper](https://arxiv.org/pdf/1303.5778.pdf) - which discusses RNNs, LSTMs, Stacked RNNs, and even Bidirectional RNNs - a one-layer RNN is built as follows (eq 1 and 2):
  $$\begin{align}
  h\_t &= \mathcal H(W\_{xh} x\_t + W\_{hh}h\_{t-1}+b\_h)\\
  y\_t &= W\_{hy}h\_t + b\_y
  \end{align}$$
  These two lines should be iterated for the sequence length $T$, so you get as output hidden states $h=(h\_1,...,h\_T)$ and sequences $y=(y\_1,...,y\_T)$. $\mathcal{H}$ is the activation function.
  
  According to the paper:
  
  > Deep RNNs can be created by stacking multiple RNN hidden layers on top of each other, with the output sequence of one layer forming the input sequence for the
  > next.
  
  This means that the recurrence is at the level of computing the hidden state, $h\_t$, meaning that if you have $N$ recurrent layers you compute all the $h^n$ sequences iteratively, and then when at the end you compute the output sequence $y$ which is also the network's output. See Eq. 11 and 12.
  
  In general, I've seen that RNNs have different implementations. For example, in Keras the recurrence occurs on the output sequence $y$: so you compute all the output sequences, and optionally also return the states (if so these will inizialize the states of the next layer), which are fed to the next recurrent layer.
  
  
  ---
  
  