I am using DistributedDataParallel on a single machine with multiple GPUs, and I’m having trouble collecting loss and accuracy between GPUs.
I have each process (GPU) printing the loss and accuracy of its training, but I want to track the overall loss and accuracy. I have tried using multiprocessing.RLock as an argument to torch.multiprocessing.spawn, but this fails.
What is the best way to collect the results when training with DistributedDataParallel?