If your module has a lot of parameters, wrapping it in nn.DataParallel might create a slowdown since these parameters have to be scattered.
Have a look at the ImageNet example 15, where only model.features is wrapped in data parallel. This is also explained in the One weird trick paper 10.
In your use case you could try to leave the embedding layer on a single device.