you can look at https://github.com/OpenNMT/PyOpenNMT 350 to see how to wrap RNN in DataParallel
You have to check whether you are using batchFirst for your RNN and also which dimension is being scattered by DataParallel