Hello everybody!
I’m currently implementing a recurrent PPO policy and as of now it works for gym-minigrid-hallway and CartPole (masked velocity). However, I’ve got quiet some open questions left that leave me in doubts about my implementation.
In comparison to a none-recurrent PPO implementation, I just added the hidden states of a GRU layer to the agents’ experience tuples. Concerning the sampling of mini batches, I make sure that sequences are maintained for each agent. That are all the changes I’ve made.
One odd thing is that one PPO cycle/update does not take longer to compute while using the recurrent policy. I expected it to be notably more expensive. This raises the question to me whether the sequence of hidden states are back-propagated correctly. How could I check/verify that the gradients flow into the whole sequence of experience tuples, while not flowing into a new episode?
For reference, these are all the open questions and a link to my implementation 3.