nn.DataParallel will reduce all parameters to the model on the default device, so you could directly store the model.module.state_dict().
If you are using DistributedDataParallel, you would have to make sure that only one rank is storing the checkpoint as otherwise multiple process might be writing to the same file and thus corrupt it.
Here 54 is an example how to do so.
CC @Janine