@tommy19970714
To be honest I’ve found success using the default parameters on my use-case so I never got the chance to dive deeper into the deepspeed configuration, but  I found this that might be helpful to you.

  
      github.com/huggingface/transformers
  
  
    
  
	  
  

  
    
      OOM when trying to fine tune patrickvonplaten/led-large-16384-pubmed 16
    

    
      
        opened 
        
          
        
        Feb 5, 2021
      
      


      
        
          
          mmoya01
        
      
    
  



  I'm currently following this notebook but instead I'm using patrickvonplaten/led-large-16384-pubmed
tokenizer = AutoTokenizer.from_pretrained("patrickvonplaten/led-large-16384-pubmed",)
led = AutoModelForSeq2SeqLM.from_pretrained(
 "patrickvonplaten/led-large-16384-pubmed",
 gradient_checkpointing=True,
 use_cache=False,
)
instead of allenai/led-large-16384 as the...



    DeepSpeed