Hey,
That’s exactly the question I needed confirmation on. However, I tried initiating h0 as zeros at the beginning of each episode and make the agent pass it from one step to another until episode end. But the agent didn’t learn. Not even after couple million frames on any of the Atari games.
I must be doing something wrong!
On the second point, the output is nothing but hidden states from all sequences stacked. i.e. last value of output == h_n
If that is the case why not pass h_n to the dense layer instead of output. it’s computationally cheaper and backpropogation will go through the other hidden layers by default since h_n was calculated from them. What do you think?