I have 4xP100s and saved an initial checkpoint after nn.DataParallel(model) but when I tried to load it in another model I got an error
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/gpfs/alpine/world-shared/gen011/shubhankar/summitdev/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/gpfs/alpine/world-shared/gen011/shubhankar/summitdev/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/gpfs/alpine/world-shared/gen011/shubhankar/summitdev/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 146, in forward
    "them on device: {}".format(self.src_device_obj, t.device))
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1