pl3:

Slightly unclear what you mean here. I had the same batch size for each training loop which meant that each GPU in the DP would have been receiving 1/4th the batch size which I was assuming should have been faster.


I am not familiar with fastai’s DDP wrapper. When using the raw DDP API, applications need to spawn one process per GPU and then create one DDP instance and one dataloader in each process. With this setting, the per-process dataloader should use batch_size/world_size as the new batch size.
Given the linked code, looks like it does not spawn subprocess for you. And it only calls init_process_group when num_distrib() > 1. So, if you didn’t spawn subprocesses explicitly in application code, it might fall back to local training?

  
      github.com
  
  
    fastai/fastai/blob/3c6dca627c1f3812d58b0447bc9a45dd866c601f/fastai/distributed.py#L143-L159 1

@patch
@contextmanager
def distrib_ctx(self: Learner, cuda_id=None,sync_bn=True):
    "A context manager to adapt a learner to train in distributed data parallel mode."
    # Figure out the GPU to use from rank.  Create a dpg if none exists yet.
    if cuda_id is None: cuda_id = rank_distrib()
    if not torch.distributed.is_initialized():
        setup_distrib(cuda_id)
        cleanup_dpg =   torch.distributed.is_initialized()
    else: cleanup_dpg = False
    # Adapt self to DistributedDataParallel, yield, and cleanup afterwards.
    try:
        if num_distrib() > 1: self.to_distributed(cuda_id,sync_bn)
        yield self
    finally:
        self.detach_distributed()
        if cleanup_dpg: teardown_distrib()



  
  
    
    
  
  


In case this is helpful, here 4 is a quick example with a brief explanation of how DDP works. And this section 1 tries to explain the differences between DP and DDP.