Hey @sanchit2843
When using batch_size == 1, DataParallel won’t be able to parallelize the computation, as the input cannot be chunked by the scatter linked below.

  
      github.com
  
  
    pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L151

def forward(self, *inputs, **kwargs):
    if not self.device_ids:
        return self.module(*inputs, **kwargs)

    for t in chain(self.module.parameters(), self.module.buffers()):
        if t.device != self.src_device_obj:
            raise RuntimeError("module must have its parameters and buffers "
                               "on device {} (device_ids[0]) but found one of "
                               "them on device: {}".format(self.src_device_obj, t.device))

    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
    if len(self.device_ids) == 1:
        return self.module(*inputs[0], **kwargs[0])
    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
    outputs = self.parallel_apply(replicas, inputs, kwargs)
    return self.gather(outputs, self.output_device)

def replicate(self, module, device_ids):
    return replicate(module, device_ids, not torch.is_grad_enabled())

def scatter(self, inputs, kwargs, device_ids):



  
  
    
    
  
  



but the time of backward pass is taking 4* times in comparison to 1 V100

Can you share a minimum repro of this? In general, the 4X slowdown is possible due to Python GIL contention (which you can avoid by using DistributedDataParallel). But I feel it this should not be applicable here as  each batch only contains one sample, and the replicated models are not really involved in forward and backward.