blade:




 mrshenli:

My suspicion is that the model parameters are so light that the overhead of GIL contention, input scattering, output gathering, and model replication in DataParallel forward pass overshadows the speedup brought by multi-gpu training.


That makes sense. Let me try it on a more sophisticated code and update the answer.


So I tried on another code, with 1 GPU my code ran in 434 sec while with 2 GPUs it took 864 sec. So it shouldn’t be from the price we pay for parallelization. Also, using your line print(self.first_term(features).device) it uses all processors at each step so the code is not running in series by each GPU.