There’s a little more nuance to it if you want to control the exact GPUs that you parallelize over.  If you look at the doc for DataParallel 103 you’ll see that you can specify device_ids. If you do that, you’ll also want to make sure you load all of your variables onto the same GPU to start with (with your_variable.cuda(device_id=ID).  That should be pretty much it.