Solved by InnovArul in post #2 
              
              
                you can collect these two LSTMs into a single class and use DataParallel on it. 
Does that help? 
import torch.nn as nn
class TwoLSTMs(nn.Module):
    def __init__(self):
        self.rnn_one = nn.LSTM(input_size=10, hidden_size=20, num_layers=2)
        self.rnn_two = nn.LSTM(input_size=20, hidden_…