sbelharbi:

how to that? it isa reduce-all op over outputs.
ddp seems to focus only on synch grads…,.
what if you want to synch outputs, losses, other stuff…
for example the computed loss is not the full loss over ENTIRE minibatch, but only a chunk.
so, is there a way to gather things on demand?
still reading the doc.
i may have missed something.
but, i dont think i saw any doc about this, probably ddp does not do this but only synch grads.


DDP provides gradient synchronization across processes. If you require data be shared between processes you need to communicate between the processes Distributed communication package - torch.distributed — PyTorch 1.9.0 documentation  22.