JayanthRR:

I don’t think nn.DistrubtedDataParallel behaves differently


It depends what you are comparing it against.
DDP with multiple devides per process performs similar to nn.DataParallel, DDP with a single device per process does not and is the recommended way. Here is also a short explanation. Also here is a comparison of the distributed backends.



 JayanthRR:

What should I do, if I want to concatenate the outputs of the forward passes across all gpus while using nn.DistributedDataParallel ?


You don’t have to do it manually and the output of the DDP model should already contain all samples. Have a look at this tutorial 2 for an example.