Answers.Body: <p>The second implementation looks more correct and inline with how Bidirectional is defined. Specifically, bidirectionality doen't change the forward/backward logic of either direction, and just merges (concat/sum/...) the outputs of forward/backward at a matching timestep <code>t</code>.</p>
  
  <p>You can check how Keras implements it <a href="https://github.com/keras-team/keras/blob/7a39b6c62d43c25472b2c2476bd2a8983ae4f682/keras/layers/wrappers.py#L388" rel="nofollow noreferrer">here</a>. There are distinct <code>self.forward_layer</code> and <code>self.backward_layer</code> that are initialized separately.</p>
  
  <p>Your <code>for</code> loop doesn't look ok though. To calculate the output at time step <code>0</code> you'd have to calculate <code>forward(0)</code> and <code>backward(n)</code>, which means you have to run backward on all the samples first. In practice, each direction is calculated separately and the results are merged afterwards. Check the implementaion in Keras <a href="https://github.com/keras-team/keras/blob/7a39b6c62d43c25472b2c2476bd2a8983ae4f682/keras/layers/wrappers.py#L529" rel="nofollow noreferrer">here</a>. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-07T08:06:38.170Z
Answers.LastActivityDate: 2020-04-07T08:06:38.170Z
Answers.UserDisplayName: olix20
Body: <p>I'm trying to understand how Bidirectional RNNs work.</p>
  
  <p>Specifically, I want to know whether a single cell is used with  different states, or two different cells are used, each having independent parameters.</p>
  
  <p>In pythonic pseudocode,</p>
  
  <p>Implementation 1:</p>
  
  <pre><code>cell = rev_cell = RNNCell()
  cell_state = cell.get_initial_state()
  rev_cell_state = rev_cell.get_initial_state()
  for i in range(len(series)):
      output, cell_state = cell(series[i], cell_state)
      rev_output, rev_cell_state = rev_cell(series[-i-1], rev_cell_state)
      final_output = concatenate([output, rev_output])
  </code></pre>
  
  <p>Implementation 2:</p>
  
  <pre><code>cell = RNNCell()
  rev_cell = RNNCell()
  cell_state = cell.get_initial_state()
  rev_cell_state = rev_cell.get_initial_state()
  for i in range(len(series)):
      output, cell_state = cell(series[i], cell_state)
      rev_output, rev_cell_state = rev_cell(series[-i-1], rev_cell_state)
      final_output = concatenate([output, rev_output])
  </code></pre>
  
  <p>Which of the above implementations is correct? Or is the working of Bidirectional RNNs completely different altogether?</p>
  
Tags: <deep-learning><recurrent-neural-networks><long-short-term-memory><time-series><gated-recurrent-unit>
Title: Inner working of Bidirectional RNNs
UserDisplayName: Susmit Agrawal
ThreadText: # Inner working of Bidirectional RNNs
  
  (Asked by: username_0 on 2020-04-06)
  
  I'm trying to understand how Bidirectional RNNs work.
  
  Specifically, I want to know whether a single cell is used with different states, or two different cells are used, each having independent parameters.
  
  In pythonic pseudocode,
  
  Implementation 1:
  
  ```
  cell = rev_cell = RNNCell()
  cell_state = cell.get_initial_state()
  rev_cell_state = rev_cell.get_initial_state()
  for i in range(len(series)):
      output, cell_state = cell(series[i], cell_state)
      rev_output, rev_cell_state = rev_cell(series[-i-1], rev_cell_state)
      final_output = concatenate([output, rev_output])
  
  ```
  
  Implementation 2:
  
  ```
  cell = RNNCell()
  rev_cell = RNNCell()
  cell_state = cell.get_initial_state()
  rev_cell_state = rev_cell.get_initial_state()
  for i in range(len(series)):
      output, cell_state = cell(series[i], cell_state)
      rev_output, rev_cell_state = rev_cell(series[-i-1], rev_cell_state)
      final_output = concatenate([output, rev_output])
  
  ```
  
  Which of the above implementations is correct? Or is the working of Bidirectional RNNs completely different altogether?
  
  **Tags:** deep-learning, recurrent-neural-networks, long-short-term-memory, time-series, gated-recurrent-unit
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-07. Score: 2 | Accepted answer
  
  The second implementation looks more correct and inline with how Bidirectional is defined. Specifically, bidirectionality doen't change the forward/backward logic of either direction, and just merges (concat/sum/...) the outputs of forward/backward at a matching timestep `t`.
  
  You can check how Keras implements it [here](https://github.com/keras-team/keras/blob/7a39b6c62d43c25472b2c2476bd2a8983ae4f682/keras/layers/wrappers.py#L388). There are distinct `self.forward_layer` and `self.backward_layer` that are initialized separately.
  
  Your `for` loop doesn't look ok though. To calculate the output at time step `0` you'd have to calculate `forward(0)` and `backward(n)`, which means you have to run backward on all the samples first. In practice, each direction is calculated separately and the results are merged afterwards. Check the implementaion in Keras [here](https://github.com/keras-team/keras/blob/7a39b6c62d43c25472b2c2476bd2a8983ae4f682/keras/layers/wrappers.py#L529).
  
  
  ---
  
  