Hi,
In your example, you use the hidden state sequentially for each network one after the other. Is that what you want? Because if you want that, then you cannot really run them in parallel on different GPUs as they will need to wait for the previous one to finish.