Can you try to profile the forward pass of your model? I am wondering why it will be slower. One possibility is that because you are making K calls to nn.LSTM's forward for K layers, the CUDA kernels might be launched separately everytime, while if you used the num_layers argument, that overhead would not be there. However, I am not sure of this, so maybe a profiling would identify any other bottlenecks.