I think if each DQN learner’s batch is big enough, the overall speed will be much faster even if they have to lose some cycles sending the gradient and downloading updates from the parameter server. That’s what Deepmind did in their GORILA paper 7. “Algorithm 1” on page 5.
I can imagine how the code would be written in Tensorflow’s async framework. But it’s not obvious how to translate into PyTorch’s distributed primitives.