Hi, if you’re within a single node you can probably coordinate these updates with something like a multiprocessing.Manager, see multiprocessing — Process-based parallelism — Python 3.9.1 documentation 13.
Alternatively, if your training is across different nodes entirely, you have a few options:

Have a per-node data pool, and just proceed with the above approach
Have a global data pool that is replicated across each node with the MP manager. You can then probably use pytorch APIs such as dist.broadcast_object_list and dist.scatter_object_list to share the required data.