Thank you for the suggestion! So far I’m satisfied with single-machine, multi-gpu setting. I did a bit of research and here’s a good post on how to use mp.Manager: https://stackoverflow.com/questions/10415028/how-can-i-recover-the-return-value-of-a-function-passed-to-multiprocessing-proce 13, in case anyone else is interested. BTW I should point out that torch.multiprocessing is just a thin layer of wrapping around Python’s native multiprocessing, so most stuff are similar and can be used directly in the same style.
Basically a dict can be shared among processes spawned, which suits my case. I guess any pickable primitives like List will also do, though.
Will be back and update this thread once I’ve extended to a multi-machine setting.