Hmmm So basically it does a copy of the model in each gpu. Then, when you call forward, it takes the batch, created as many chunks as gpus a send them to the corresponding one.
it assumes the batch dimension is the dimension 0. I think this issue can happens due to he fact rnn operates as temporal dimension first. Soo how is the input to your network sorted?
WRT DistributedDP I don’t really know why. It wasn’t the case some versions ago. It seems the performance of DP is worse (https://pytorch.org/docs/stable/notes/cuda.html#cuda-nn-ddp-instead 1)
Anyway both will probably work in a similar way.