I had run into the same problem when trying to combine rnn modules with DataParallel.
If you wrap the rnn in a module where the forward function only requires an input parameter, it works fine. It doesn’t seem to work when you need both the input and hidden parameter. If you can contain your hidden parameter logic within your module, it’s an effective work-around.