DataParallel is not working for me over multiple GPUs with batch_first=False, and I think there are other questions in the forum with similar issues iirc. The two snippets I posted above (GRU  and LSTM) will not work with multiple GPUs even when splitting on a different dimension with batch_first=False (I made the snippets self-contained to make it easy to verify). It seems from other questions here that batch_first=True works fine, but I don’t think it works with False unless my code is wrong --which is entirely possible. If you have a minute I’d appreciate the validation of the code as I’m learning pytorch and can’t say for sure.