This was so long ago I dont remember if I was able to solve it. Last time I used policy gradients I went with the hogwild style of training, where instead I share the weights over multiple processes and each process can update the weights asynchronously.
Another idea would be to use rpc. The only experience I have is with gRPC where each servicer can run in its own process, maybe that can be used in a similar way? https://pytorch.org/docs/stable/rpc.html 1