Why isn’t it possible? Each worker can send its gradient to the central parameter server without first waiting for the other workers and then do all_reduce. The Tensorflow example code shows just that.