Hey @bapi

For DataParallel, somewhere it is mentioned that at present it does not run on more than 8 GPUs;

Just curious, could you please point me to the doc with this claim? This is new to me, I wasn’t aware there is such a limitation in DP.

however, I could not find similar info about DDP (I may have missed it).

We recently tested DDP using 256 GPUs, and it runs fine. Could this error be sth specific to the imagenet example? cc @fmassa for vision questions

Moreover, as all the processes load their own module locally on each of the devices without a broadcast during initialization, is not it unexpected?

There is a broadcast in DDP ctor. Please see the link below:

  
      github.com
  
  
    pytorch/pytorch/blob/c71ec1c717e5b225f28ef3bacde416c69f3c4d77/torch/nn/parallel/distributed.py#L324-L329 2

# Sync params and buffers
module_states = list(self.module.state_dict().values())
if len(module_states) > 0:
    self._distributed_broadcast_coalesced(
        module_states,
        self.broadcast_bucket_size)