nn.DataParallel will allocate the model, inputs, and outputs on the default device (or whichever is used in the to() operation) and will then scatter and gather the parameters, data, gradients etc. as described in this blog post 5.
The communication overhead could cause a slowdown compared to DistributedDataParallel using a single process per GPU, which is why we generally recommend to use the latter approach.