TT_YY:

However, I could not find an article explaining whether or not loss is also reduced.


No, loss is not reduced because there is only one loss tensor with DP. Besides, the gradients are actually accumulated automatically by the autograd engine. As DP is single-process-multi-thread, all threads share the same autograd engine, and hence ops on different threads will be added to the same autograd graph.

Is the loss obtained by above code averaged over all the GPUs, which is exactly same as a loss computed by a serial program? Or, is the loss a value from just one GPU (gpu0)? I need to plot a loss chart, so I wonder if the loss is averaged over the GPUs.

DP’s forward function will gather all outputs to cuda:0 (by default) and then return the gathered result. So, in the code above outputs is on one GPU and hence loss is also on one GPU.

The same question applies to outputs. I also need to compute training accuracy using outputs in above code. Does it hold the results of all the GPUs? If so, in what structure of a tensor are they stored?

Below is DP’s forward function. The outputs var on line 161 holds the output on different GPUs, but the gather function on line 162 copied them to one GPU.

  
      github.com
  
  
    pytorch/pytorch/blob/fa6b34b54c731938327c8e30e08b287a10b86b0a/torch/nn/parallel/data_parallel.py#L147-L162 9

def forward(self, *inputs, **kwargs):
    if not self.device_ids:
        return self.module(*inputs, **kwargs)

    for t in chain(self.module.parameters(), self.module.buffers()):
        if t.device != self.src_device_obj:
            raise RuntimeError("module must have its parameters and buffers "
                               "on device {} (device_ids[0]) but found one of "
                               "them on device: {}".format(self.src_device_obj, t.device))

    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
    if len(self.device_ids) == 1:
        return self.module(*inputs[0], **kwargs[0])
    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
    outputs = self.parallel_apply(replicas, inputs, kwargs)
    return self.gather(outputs, self.output_device)



  
  
    
    
  
  


If you want to access individual output on different GPUs, you can do so in the forward function of your model (the one you passed to DP ctor). E.g.,
class MyModel(nn.Module):
  def __init__(self):
    self.fc = nnLinear(10, 10)
  def forward(self, input):
    output = self.fc(input)
    print("per-GPU output ", output)
    return output


dp = DataParallel(MyModel())
outputs = dp(inputs) # this outputs is on one GPU


Regarding DDP, above codes are written in each process running on respective GPU. In this case, how can I access the values on all the GPUs to plot the averaged loss and total accuracy?

You can use gather 38 or all_gather 27 or all_reduce 35 to communicate the loss to one process and print it.
BTW, could you please add a “distributed” tag to distributed training related questions? People working on distributed training monitor that tag and can get back to you promptly.