Thanks for your sharing! My understanding is that the parameter updating is done after the gradient is reduced to the GPU-1(the GPU which scatters replica and inputs), and the parameter updating only takes place on GPU-1. Please correct me if wrong.