Hi, thank you for your help and patience. May I ask if it works in DistributedDataParallel as well? Since the doc 1 reads

Besides, when loading the module, you need to provide an appropriate  map_location  argument to prevent a process to step into others’ devices. If  map_location  is missing,  torch.load  will first load the module to CPU and then copy each parameter to where it was saved, which would result in all processes on the same machine using the same set of devices.