You could check, if your GPU has any resources left for a parallel execution of the RNNs.
RNNs are using matrix multiplications internally, which are executed via cublas on the GPU. Generally matmuls tend to saturate the device as much as possible, so that other kernels might have to wait for their execution. You could check the profile via e.g. Nsight Systems and also take a look at e.g. this post.