@Maimonator Can you tell me how to set the parameter of shared_ddp?
I have tried to use deepspeed. But I occurred error.
That error is the same as the following issue (RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one...).
Can you tell me your json of deepspeed config?

  
      github.com/huggingface/transformers
  
  
    
  
	  
  

  
    
      RuntimeError: while running run_common_voice.py (XLSR wav2vec finetuning week) 10
    

    
      
        opened 
        
          
        
        Mar 24, 2021
      
      


      
        
          
          raja1196
        
      
    
  



  Environment info

transformers version: 4.5.0.dev0 (I tried running it on 4.4.0 as well, gave the same error)
Platform: Ubuntu (running on a virtual...