I understand, thanks for the clarification.
You can use existing torch.distributed primitives gather or all_gather to get all results to a single or all processes, respectively. You say you’re outputting dictionaries, so you can’t do it with functions in core yet, and would need to serialize the dictionaries yourself. Coincidentally, @mrshenli and I were talking about adding this to core yesterday, and he created an issue to track it: https://github.com/pytorch/pytorch/issues/23232 42. This doesn’t solve your issue right now though.
To solve this today, I would indeed write everything to a shared filesystem if you have one (with torch.save), probably named after the rank of the process, run a barrier to ensure all writes are done, and then torch.load all of them on the process where you want to collect the results.