Thanks for the update, Albert.
A lot of water has passed under the bridge since this discussion. We have integrated bf16 support in HF Transformers and even have benchmarks to show its performance under RTX-3090 and A100

  

      github.com/huggingface/transformers
  

  
    
  
	  
  

  
    
      [Benchmark] HF Trainer on RTX-3090 9
    

    
      
        opened 
        
          
        
        Dec 3, 2021
      
      


      
        
          
          stas00
        
      
    

    
        
          Benchmarks
        
        
          WIP
        
    
  


  
    # 🖥 Benchmarking RTX-3090 `transformers` w/ HF Trainer:

We are going to use a… special benchmarking tool that will do all the work for us. https://github.com/huggingface/transformers/pull/14934

This is the index post and specific benchmarks are in their own posts below:

1. [fp16 vs bf16 vs tf32 vs fp32](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004390803)
2. [gradient accumulation steps](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537)
3. [gradient checkpointing](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004422281)
4. [batch size](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537)
5. [optimizers](https://github.com/huggingface/transformers/pull/14744#issuecomment-1002398095)

TODO:
- redo optimizers once https://github.com/huggingface/transformers/pull/14744 gets merged
- a combined benchmark with the slowest vs fastest configs
- other suggestions?
  

  

  
    
    
  

  


  

      github.com/huggingface/transformers
  

  
    
  
	  
  

  
    
      [WIP] [Benchmark] HF Trainer on A100 7
    

    
      
        opened 
        
          
        
        Jan 4, 2022
      
      


      
        
          
          stas00
        
      
    

    
    
  


  
    # 🖥 Benchmarking A100 `transformers` w/ HF Trainer:

We are going to use a spe…cial benchmarking tool that will do all the work for us. https://github.com/huggingface/transformers/pull/14934

This is the index post and specific benchmarks are in their own posts below:

Please ignore for now as I'm gathering results.


This is the index post and specific benchmarks are in their own posts below:

1. [fp16 vs bf16 vs tf32 vs fp32](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004543189)
2. ...