If you are using the LSTM or GRU layer, you just feed the initial hidden state. The layer propagates the sequences and the consecutive hidden states by itself.
I published a reference/baseline implementation for PPO that shows each important detail of a recurrent policy.
  
      

      GitHub
  

  
    

GitHub - MarcoMeter/recurrent-ppo-truncated-bptt: Baseline implementation of... 3

  Baseline implementation of recurrent PPO using truncated BPTT - GitHub - MarcoMeter/recurrent-ppo-truncated-bptt: Baseline implementation of recurrent PPO using truncated BPTT