You can use torch.multiprocessing: http://pytorch.org/docs/master/notes/multiprocessing.html 3
You can find a clean example of a single model updated by independent workers on different GPUs here: https://github.com/ikostrikov/pytorch-a3c 6, it’s about reinforcement learning, but I think you should easily adapt it to train your RNN