The faster GPU processes will wait for other GPUs to finish their backward computation.
By default, DDP synchronizes gradients and parameters and then performs the next forward computation.
The differing speed case is quite common in practice.
You can take a look at the forward function of DDP.

  
      github.com
  
  
    pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L675 2

# Calling _rebuild_buckets before forward compuation,# It may allocate new buckets before deallocating old buckets# inside _rebuild_buckets. To save peak memory usage,# call _rebuild_buckets before the peak memory usage increases# during forward computation.# This should be called only once during whole training period.if self.reducer._rebuild_buckets():    logging.info("Reducer buckets have been rebuilt in this iteration.")if self.require_forward_param_sync:    self._sync_params()if self.ddp_uneven_inputs_config.ddp_join_enabled:    # Notify joined ranks whether they should sync in backwards pass or not.    self._check_global_requires_backward_grad_sync(is_joined_rank=False)if self.device_ids:    if len(self.device_ids) == 1:        inputs, kwargs = self.to_kwargs(inputs, kwargs, self.device_ids[0])        output = self.module(*inputs[0], **kwargs[0])    else: