This answer might be helpful,

  
    
    
    Multi layer RNN with DataParallel 
  
  
    Here is one solution for using nn.DataParallel I find works well. It can return both rnn output and hidden states from your module, using batch_first = False mode (which is a popular mode). 


Use batch_first = False for seq2seq modeling (e.g., encoder-decoder architecture), which seems to the be the popular way for feeding the input data, so we get input batch with shape (max_sequence_length, batch_size, num_embeddings). 


Then set dim=1 to scatter the data the along the dimension for batches …