vadimkantorov:

Will all launched processes do CUDA init?


CUDA is lazily initialized. So if one process is not touching a specific device, corresponding CUDA context shouldn’t be created on that device.

Is it safe?

Even if multiple processes create context on the same device, it won’t crash, but each context consumes about 500MB CUDA memory, which is not desired.

Should we explicitly set CUDA_VISIBLE_DEVICES per launched process to ensure that it can see only one device?

This is the recommended way, as it also rules out the possibility that some third-party library accidentally create tensors on other devices.

Would it prevent it from rpc using nccl?

Which rpc are you referring to? Are you using DDP in conjunction with torch.distributed.rpc?

Is it sensible to do a forced parameter sync once in a while? Inherent GPU parallelism non-determinism can cause parameter divergence that could cause replica parameter divergence for some sensitive models (even if gradients are sychronized)?

If there are drifts, then, yes, manually sync once a while would help.

How does one do it?

You can use broadcast, see the code linked below. If you would like to calculate average, you can also use all_reduce and then divide by world_size.

  
      github.com
  
  
    pytorch/pytorch/blob/3806c939bda0df4c0f38b9d356c004f384535ac1/torch/nn/parallel/distributed.py#L404 2

    self.bucket_bytes_cap = int(bucket_cap_mb * 1024 * 1024)

    # Sync params and buffers
    self._sync_params_and_buffers(authoritative_rank=0)

    self._ddp_init_helper()

def _sync_params_and_buffers(self, authoritative_rank=0):
    module_states = list(self.module.state_dict().values())
    if len(module_states) > 0:
        self._distributed_broadcast_coalesced(
            module_states,
            self.broadcast_bucket_size,
            authoritative_rank)

def _ddp_init_helper(self):
    """
    Initialization helper function that does the following:

    (1) replicating the module from device[0] to the other devices
    (2) bucketing the parameters for reductions



  
  
    
    
  
  



When does it make sense to do NUMA node pinning? CPU affinity pinning?

Hey @ptrblck, do you know what’s the best practice here?