In replacing DataParallel with DistributedDataParallel, I encountered that each epoch compute a number of metrics. While using DataParallel this still was fine, as everything was running in the same process.
However, I would like to collect the metrics on the first GPU as well. There metric could either be computed on the GPU or CPU. As I understand, with DistributedDataParallel in case there are 2 GPUs,  3 processes are started (one to collect things). Then, loss.backward() should work as expected. But how do I do this for my metrics?