Yes, as you can see in the ImageNet example, you can just use DataParallel on a certain part or your model, which will work fine.
In fact you can split your model in any way, e.g. leaving one part on the CPU, another on a single GPU, and again another in DataParallel.
As long as you push the activations and inputs to the appropriate device, everything will work fine.
Otherwise, you’ll get some errors.