IMHO, By default, LSTM is a sequential architecture. Whatever parallelism that you achieve by nn.DataParallel is to speed it up by using multiple-GPUs. i.e., for higher batch sizes, but it is still sequential.