jteuwen:

As I understand, with DistributedDataParallel in case there are 2 GPUs, 3 processes are started (one to collect things). Then, loss.backward() should work as expected. But how do I do this for my metrics?


There is no separate process to collect things. You are responsible for starting all processes, either through running them yourself from a terminal, through torch.distributed.launch, or with some other runner such as mpirun. The typical mode of execution is to use 1 GPU per process.