How to use DataParallel:
model = DataParallel(model, dim=your batch dim in input, device_ids=[main_id, other_ids …], output_device=main_id)
Note that main_id (GPU that store the original model parameter) should be the first in the list of device_ids.
You can use DataParallel since it’s easier to setup and test, but remember to:

Set the batch dimension of DataParallel. The default is dim=0 but sometime you might want to apply another dimension.
(e.g. my input size is (Time, Batch, Dim_Data), and the model require a fully time series. In this scenario I will apply dim=1 instead of dim=0, because If I choose dim=0, the time serie will be split into multiple fragments.)
DataParallel return a wrapped model, so use the wrapped model to forward instead of original one, and remember to…
…handle the state_dict of wrapped model before save the state_dict into files. DataParallel will append prefix “module.” to each key of the original state_dict.keys(), you have to remove the prefix before saving the state_dict.
Set the GPU with larger memory as output_device , and also pass model parameter and your input to this GPU. output_device need to store both data and model parameters, so larger GPU memory is favorable.