Solved by mrshenli in post #2 
              
              
                Hey @sakh251 

1- The data should be sent to the parameter server (for a large dataset is it possible???) 
2- it Async 
3- distributed auto grad has been used 
am I correct? 

Yep, you are correct. 

I am wondering is there any way to implement a sync parameter server with the RPC framework without …