Hi,
I am wondering if there is a way to share gradients in multiprocessing.
Currently, I am following the [hogwild] (http://pytorch.org/docs/master/notes/multiprocessing.html#hogwild 37) example to run the asynchronous training. The shared model is the CPU. The local model is on the GPU and has their own or shared optimizer. Each local model can update the shared model directly.
I want to modify the current case to the synchronous version. For example, each local model can upload their gradients to the shared model. After aggregating a certain amount the gradients, the shared model can update itself.  However, in this example [hogwild] (http://pytorch.org/docs/master/notes/multiprocessing.html#hogwild 37), the gradient part of the shared model is not shared with other processes.