Hi,
About your question:

Yes, RNNs can benefit from optimized GPU implementations, and PyTorch wraps cudnn, which gives even further speedups
Yes, it will run out of the box, but only in one GPU. If you want to parallelize over multiple GPUs, check http://pytorch.org/docs/master/nn.html#torch.nn.DataParallel 63 for a simple way to distribute computations batch-wise over multiple GPUs
GPUs shine compared to CPUs for larger batch sizes.
If you use nn.DataParallel, everything is handled for you automatically. But you might want to have different ways of using multiple GPUs (for example parts of one model in GPU1, and other parts in GPU2), in which case you need to ship the different parts to the different GPUs yourself (via result.cuda(gpu_idx)).

nn.DataParallel already gathers the information from multiple GPUs for you 20

For small models, you won’t see any benefits from using GPUs over CPUs, and it won’t improve if you use multiple GPUs. You will need to increase the model size to start to see improvements, because there is some communication overhead to transmit data from different GPUs. Also, there are a number of tricks that are used for improving multi-GPU usage, see https://arxiv.org/abs/1404.5997 38 for example.

Hope this helps!