Answers.Body: <p>I have not implement the backprop of a bi-directional RNN from scratch so I can't be sure my answer is correct but I hope it helps.</p>
  <p>You can see how bi-directional RNN works from <a href="https://coursera.org/share/4de91466d7926b472263bf4157d73e94" rel="nofollow noreferrer">this</a> video from Andrew NG. I got the image below from that video:
  <a href="https://i.sstatic.net/GrpCg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GrpCg.png" alt="enter image description here" /></a></p>
  <p>For more clarity:
  <a href="https://i.sstatic.net/fxCbG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fxCbG.png" alt="enter image description here" /></a></p>
  <p>So if you know how to backprop through a simple RNN, you should be able to do so for bi-directional RNN.</p>
  <p>If you need more detail, let me know.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-19T13:17:57.137Z
Answers.Text: Thank you @amin. I have one more doubt as to if I decide to fetch output from intermediate layers of the bi-directional GRU ( which is in this case the same as hidden state as GRU outputs only the hidden state at any time 't', how do i extract the weights of such a layer which I want to fetch?
Answers.UserDisplayName: rahul tomar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-20T02:54:53.477Z
Answers.Text: Unfortunately I can't understand what you wrote well. But I will tell you 2 notes and I hope they help: 1- if you are using keras, you can set the `return_sequences` parameter of your GRU layer to true, and then when using GRU you will get a list of outputs containing output of each timestep. 2- RNN will use just one bunch of weights for all timesteps (so the same weights are applied in each timestep) and BRNN will use two bunch of weights, one for forward (going from x<1> to x<4>) and one for backward (going from x<4> to x<1> in the picture above)
Answers.UserDisplayName: amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-20T04:29:31.653Z
Answers.Text: Thank you @amin. I will just like to add one more scenario to the one you suggested in the second point. Suppose I stack BRNN layers in a model I am trying to build. For ex- if no. of stacked layers are 5, and I want to extract intermediate outputs of BRNN layer numbers- 2, 3, 4 apart from ( with the inputs fed to layer 1 of BRNN). I am using Keras and Functional API to implement. In this case, as I am extracting outputs from inner BRNN layers, will the inner connections of my network differ from a normal BRNN? Moreover how do i extract intermediate weights when utilizing backpropagation?
Answers.UserDisplayName: rahul tomar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-20T06:53:06.300Z
Answers.Text: @rahultomar I'm glad if I helped. I prefer to give you a useful links to understand how RNNs work in keras and how to use it. I guess if you understand what has been said in this link, you can answer the questions above. If you can't get your answer by looking at this link, I suggest you to have a post in stackoverflow (as ai.stackexchange is not related to the programming problems). [Keras - Working with RNNs](https://keras.io/guides/working_with_rnns/). Good luck and have fun programming
Answers.UserDisplayName: amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-20T09:09:40.390Z
Answers.Text: Thank you @amin for your response. As suggested i will post on the required forum. Meanwhile, coming to the same weights (one for forward and another for backward which are shared for the BRNN). Does this hold true as we stack BRNN layers on top of each other? More specifically, if W1 are the set of weights in the forward RNN of first BRNN layer while W2 are weights of backward RNN of first layer. As both W1 and W2 hold for all timesteps in the same layer, do they change as we stack up more layers or they remain same ?
Answers.UserDisplayName: rahul tomar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-27T06:41:40.277Z
Answers.Text: @rahultomar I saw your comment late. When you stack them on top of each other, you will get new weights and biases.
Answers.UserDisplayName: amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T15:09:18.003Z
Answers.LastActivityDate: 2020-10-11T15:09:18.003Z
Answers.UserDisplayName: amin
Body: <p>I am aware that back-propagation through time is used for training the recurrent neural network. But I am not able to understand how this happens for the bi-directional versions of the recurrent neural networks?</p>
  <p>So, I was hoping if anyone help me with:</p>
  <ol>
  <li>Understanding with an example the training of bi-directional recurrent neural networks using back-propagation through time? (I tried following the original paper <a href="https://ieeexplore.ieee.org/document/650093" rel="nofollow noreferrer">https://ieeexplore.ieee.org/document/650093</a>, but it was kind of confusing for me when they perform the backward pass for training)</li>
  </ol>
  
Tags: <machine-learning><deep-learning><training><recurrent-neural-networks><backpropagation>
Title: How does back-propagation through time work for optimizing the weights of a bidirectional RNN?
UserDisplayName: rahul tomar
ThreadText: # How does back-propagation through time work for optimizing the weights of a bidirectional RNN?
  
  (Asked by: username_0 on 2020-10-10)
  
  I am aware that back-propagation through time is used for training the recurrent neural network. But I am not able to understand how this happens for the bi-directional versions of the recurrent neural networks?
  
  So, I was hoping if anyone help me with:
  
  1. Understanding with an example the training of bi-directional recurrent neural networks using back-propagation through time? (I tried following the original paper <https://ieeexplore.ieee.org/document/650093>, but it was kind of confusing for me when they perform the backward pass for training)
  
  **Tags:** machine-learning, deep-learning, training, recurrent-neural-networks, backpropagation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-11. Score: 1
  
  I have not implement the backprop of a bi-directional RNN from scratch so I can't be sure my answer is correct but I hope it helps.
  
  You can see how bi-directional RNN works from [this](https://coursera.org/share/4de91466d7926b472263bf4157d73e94) video from Andrew NG. I got the image below from that video:
  [![enter image description here](https://i.sstatic.net/GrpCg.png)](https://i.sstatic.net/GrpCg.png)
  
  For more clarity:
  [![enter image description here](https://i.sstatic.net/fxCbG.png)](https://i.sstatic.net/fxCbG.png)
  
  So if you know how to backprop through a simple RNN, you should be able to do so for bi-directional RNN.
  
  If you need more detail, let me know.
  
  > **Comments:**
  > **username_0**: Thank you @username_1. I have one more doubt as to if I decide to fetch output from intermediate layers of the bi-directional GRU ( which is in this case the same as hidden state as GRU outputs only the hidden state at any time 't', how do i extract the weights of such a layer which I want to fetch?  
  > **username_1**: Unfortunately I can't understand what you wrote well. But I will tell you 2 notes and I hope they help: 1- if you are using keras, you can set the `return_sequences` parameter of your GRU layer to true, and then when using GRU you will get a list of outputs containing output of each timestep. 2- RNN will use just one bunch of weights for all timesteps (so the same weights are applied in each timestep) and BRNN will use two bunch of weights, one for forward (going from x<1> to x<4>) and one for backward (going from x<4> to x<1> in the picture above)  
  > **username_0**: Thank you @username_1. I will just like to add one more scenario to the one you suggested in the second point. Suppose I stack BRNN layers in a model I am trying to build. For ex- if no. of stacked layers are 5, and I want to extract intermediate outputs of BRNN layer numbers- 2, 3, 4 apart from ( with the inputs fed to layer 1 of BRNN). I am using Keras and Functional API to implement. In this case, as I am extracting outputs from inner BRNN layers, will the inner connections of my network differ from a normal BRNN? Moreover how do i extract intermediate weights when utilizing backpropagation?  
  > **username_1**: @username_0 I'm glad if I helped. I prefer to give you a useful links to understand how RNNs work in keras and how to use it. I guess if you understand what has been said in this link, you can answer the questions above. If you can't get your answer by looking at this link, I suggest you to have a post in stackoverflow (as ai.stackexchange is not related to the programming problems). [Keras - Working with RNNs](https://keras.io/guides/working_with_rnns/). Good luck and have fun programming  
  > **username_0**: Thank you @username_1 for your response. As suggested i will post on the required forum. Meanwhile, coming to the same weights (one for forward and another for backward which are shared for the BRNN). Does this hold true as we stack BRNN layers on top of each other? More specifically, if W1 are the set of weights in the forward RNN of first BRNN layer while W2 are weights of backward RNN of first layer. As both W1 and W2 hold for all timesteps in the same layer, do they change as we stack up more layers or they remain same ?  
  > **username_1**: @username_0 I saw your comment late. When you stack them on top of each other, you will get new weights and biases.  
  
  ---
  
  