I want to implement the parameter server architecture. So I need to send all gradients/parameters to parameter server and average them. Then, I send back to the worker. I find the reduce operation is similar with what I want, but it is different. And all_reduce looks like a operation that each worker communicates among workers, does this setting cost more communication cost? http://pytorch.org/tutorials/intermediate/dist_tuto.html 10
Does pytorch support parameter server architecture? Parameter server and workers, like mu’s paper

  
      cs.cmu.edu
  
  
    
osdi14-paper-li_mu.pdf 5

1133.52 KB