I just recalled that hidden states are also stored as the current_state and next_state entries, and they are detached, so a single GRU layer is possible, since in this scenario, gradients will not flow through those detached hidden states. It really depends on your model implementation.
I also noticed that you have serveral different modules in the bottom PPO part, compared to that repo, may be you should also check those modules? It is also possible that you may have wrongly implemented PPO, there are many possibilities.
image519×906 17.8 KB