I finally did the code reading on DataParallel part.

  
      
      Blog – 5 Mar 19
  
  
    

How PyTorch implements DataParallel? 84

PyTorch can send batches and models to different GPUs automatically with DataParallel(model). How is it possible? I assume you know PyTorch uses dynamic computational graph. And PyTorch version is v1.0.1.