Don’t know if this is still relevant but I’ve had a similar issue using Multi-GPU so after a lot of googling I found these:
  
      huggingface.co
  
  
    

Fit More and Train Faster With ZeRO via DeepSpeed and FairScale 27



  
  
    
    
  
  


  
      
      huggingface.co
  
  
    

Trainer 16

The Trainer and TFTrainer classes provide an API for feature-complete training in most standard use cases. It’s used in most of the example scripts. Before i...


  
  
    
    
  
  


Using deepspeed helped me accelerate my training procedure, and also make sure you use shared_ddp to make sure memory is evenly distributed among GPUs.
Hopefully that helps