Ayman_Al_Jabri:

On the second point, the output is nothing but hidden states from all sequences stacked. i.e. last value of output == h_n


(Ok, first of all a disclaimer: I’m for sure no expert, but I have worked with several RL algorithms in Deep Learning)
So having this out of the way  I see what you mean, interesting question… I guess the difference is that once the linear layer only knows about the last step, in the other case it has access to the entire sequence. I think both ways are valid, but in the context with RL with unknown sequence length you will probably have to go with with the hidden state. But I also assume that you will not need any Linear non-recurrent layer, so you could probably just use the hidden state of the last layer directly as output.
Regarding the first question what algorithm are you using? DQN, DDQN, or a Policy Gradient? Which environment are you trying to solve and what is the architecture you are using?
If you are interested, I’m also happy to discuss that with our outside of that threat!