Hi everyone,
I was trying to implement a distributed training for my model and I have found to have an issue when a new checkpoint (best model) has to be saved on disk. Whenever I try to execute my training with 2 processes (one process per GPU) using DistributedDataParallel, the process with rank 1 stops without any error in the output while the master process continues to work for a while (I think then it will be stopped since the process 1 is no more able to sync).
This problem arises whenever I use the following piece of code to save my checkpoint:
        if rank == 0 and loss_trend['dev'][-1] < best_loss:
            best_loss = loss_trend['dev'][-1]
            if cfg.LOGGING.CHECKPOINTS:
                assert ckp_file, 'Checkpoint file not defined'
                state_dict = model.module.cpu().state_dict()
                torch.save(state_dict, ckp_file)
            model.cuda(gpu)
            logger.log('best model saved at: {}'.format(ckp_file))

The problem is fixed I use deepcopy on my model without touching (without moving to cpu) the original one:
        if rank == 0 and loss_trend['dev'][-1] < best_loss:
            best_loss = loss_trend['dev'][-1]
            if cfg.LOGGING.CHECKPOINTS:
                assert ckp_file, 'Checkpoint file not defined'
                state_dict = copy.deepcopy(model.module).cpu().state_dict()
                torch.save(state_dict, ckp_file)
            logger.log('best model saved at: {}'.format(ckp_file))

Since the deepcopy introduces some overhead, I want to know why it works in this case and if there are available other methods to solve my problem. Here my model is wrapped using DistributedDataParallel.
Thank you