You could try to save the state_dict of the underlying model without the nn.DataParallel wrapper as described here. Additionally, you could also push it to the CPU before saving it. This would allow you to load the model on any system later.