— sorry for possible redundancy with other threads but i didnt find an answer.
hi,
trying to do evaluation in ddp.
forward in each gpu works fine.
but how can i gather all the outputs to a single gpu (master for example), to measure metrics onces an over ENTIRE minibatch because each process forward only a chunk of the minibatch.
or we can compute the metric over each gpu, but average over single one to reduce the communication overhead, then broadcast to others if necessary also to reduce communication.
how to that? it isa  reduce-all op over outputs.
ddp seems to focus only on synch grads…,.
what if you want to synch outputs, losses, other stuff…
for example the computed loss is not the full loss over ENTIRE minibatch, but only a chunk.
so, is there a way to gather things on demand?
still reading the doc.
i may have missed something.
but, i dont think i saw any doc about this, probably ddp does not do this but only synch grads.
pytorch lightning 1 and also here 1 seems to have started working on this…
still reading their thread.
please feel free to leave a message or a snippet of code on how to do it.
the idea is to be able to evaluate using distributed computation, then gather the output or some intermediate metrics to single machine to do the average.
also, cant evaluate properly because every process have access only to a chunk of the minibatch. so, you cant loop over entire dataset unless you create a NEW dataloader, and use ddp.module to do the forward not ddp. because if you try to use ddp forward while only the master is doing the forward, it will get stuck waiting for other processes to do their forward… also, we are using distributedsampler… so it wil prevent a process from seeing the entire data.
again, i may have missed something.
thanks