Solved by mrshenli in post #2 
              
              
                Hey @aguirguis 

What confuses me is that it is not clear where does the execution (forward and backward passes) happen in this example…is it on the trainer machines or the PS machine? 

The executions scattered on both trainer and PS: 

Forward pass: input (on trainer) -> parameter (on PS) -> outpu…