Hi @MehdiZouitine,
DataParallel has a speed issue due to python GIL and its parallelization. As you mentioned, DataParallel has to recast the parameters from gpu 0 to gpu 1~3 at every forward computation.
You can try DistributedDataParallel which is faster. It is the officially recommended multi-gpu scheme.
Here’s the tutorial for DDP.

  
      pytorch.org
  
  
    

PyTorch Distributed Overview — PyTorch Tutorials 1.6.0 documentation 4



  
  
    
    
  
  


Also, you can refer to a complete training/evaluation example 1.