CDhere:

Will be back and update this thread once I’ve extended to a multi-machine setting.


For multiple machine data sharing, one option would be letting one process to serve as the data store, and use torch.distributed.rpc to push/pull data across processes.