Since your workload is tiny, you might see the overhead in the communication (scatter and gather for all GPUs).
Could you check the performance for larger models?
Also, have a look at DistributedDataParallel 25, which might perform better than nn.DataParallel.