blade:

So it is not even sharing the work between the different worker.


What is “worker” here? Do you mean GPU? If so, isn’t that expected?  IIUC, in your code, model is the DataParallel instance. So only the forward function of model would utilize multiple GPUs. See the data parallel implementation below:

  
      github.com
  
  
    pytorch/pytorch/blob/4104ab8b187d6023b0cc80b77e6944126009c532/torch/nn/parallel/data_parallel.py#L141-L156 1

def forward(self, *inputs, **kwargs):
    if not self.device_ids:
        return self.module(*inputs, **kwargs)

    for t in chain(self.module.parameters(), self.module.buffers()):
        if t.device != self.src_device_obj:
            raise RuntimeError("module must have its parameters and buffers "
                               "on device {} (device_ids[0]) but found one of "
                               "them on device: {}".format(self.src_device_obj, t.device))

    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
    if len(self.device_ids) == 1:
        return self.module(*inputs[0], **kwargs[0])
    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
    outputs = self.parallel_apply(replicas, inputs, kwargs)
    return self.gather(outputs, self.output_device)