For DQN it’s not the case, because it’s already batched. GPU makes a big speed difference over CPU on a single thread in my experiments.
I’m just looking for a way to reproduce TF’s async distributed semantics in torch.distributed rather than coming up with workarounds. It’s good to understand how to put those primitives together in a correct way, so that I have more control over the communications. There isn’t a single tutorial code on how to use torch.distributed primitives in real settings. The ImageNet example (released with v0.2) uses the nn.parallel.DistributedDataParallel wrapper whose internals are quite obscure.