Maybe, it’s because that you doesn’t set your device to load when use distributeddataparallel.
loc = 'cuda:{}'.format(args.gpu)
checkpoint = torch.load(SAVE_PATH, map_location=loc)

adding map_location option in your main_worker will solve your problem.