I want to do the exact same thing (as my multiprocesses have local models on multiple GPUs) but could not figure that out yet. There are some insights in the following link where the gradients of the local model can be copied, vica versa. I guess that it should be possible to add gradients from local model to the global shared model before making the optimizer step. Please, let me also know if you figure it out.

  
      
      GitHub
  
  
    

dchetelat/acer 24

PyTorch implementation of both discrete and continuous ACER - dchetelat/acer