Yes, all children will also be wrapped in nn.DataParallel.
You can’t specify multiple devices using cuda() or to().
As described in the tutorial you push the model onto the default device, where e.g. all gradients will be accumulated, the data will be scattered from etc.
That’s also the reason you’ll usually see a bit more memory usage on the default device.
Have a look at @Thomas_Wolf’s blog post 34 on this topic for more information.