distributed980×953 97.5 KB

I am not a Machine Learning savvy, so please mind the errors while I write this:
I think the figure will explain this right. Since I will be gathering the weights of all the networks residing at different processes, I need to pass the weight parameters right? And the weight gathering needs to happen asynchronously, at different point of time. But to get a start, at this point, we can assume that the weight gathering happens synchronously, at the same point of time.
Furthermore, the gathered weights need to be averaged at the root process and the aggregated weight should be broadcasted again to the networks for next epoch. I am not sure if you get this due to my weird way of explaining things but thanks for being modest and replying promptly.
If you could tell me a specific way to handle this, I could narrow down my scope of researching the documents and tutorials. Thank you again.