Usually “data parallel” means data operations run in parallel, but here data parallel only means that the forward passes, the fast part, have any parallel component. That’s just not very useful - as larger batches can reduce the convergence rate. So I’m using twice the power, generating twice the heat and am getting no real benefit.
The use of python makes parallelism unduly hard but one could run the autograd pass in separate processes just like hogwild, but without sharing the model weights by having each process broadcast the updates from the cost function’s backward pass. Each process would average in the updates from other GPUs.