From my point of view, you don’t have to save hidden states of each transition. Hidden states are computed based on input sequences fed to the recurrent layer.
However, it might be also possible to store hidden states along the transitions, but then you wouldn’t feed sequences and thus do not benefit from BPTT. I tried that approach for a PPO policy and workerd well for CartPole (masked velocity) and Minigrid-Hallway.