Is there any example of how to calculate the loss within multiple GPU and merge all of them later after the calculation?
Currently, we could calculate the output from a network by using DistributedDataParalel. However, the result from DistributedDataParallel is collected in device 0. Therefore, the calculation was done in 1 GPU only instead of multi-GPU.