We started compiling a wiki of how different models were pre-trained, please add your knowledge there  - thanks!

  
    
    
    Compiling data on how models were pre-trained: fp16, fp32, bf16 🤗Transformers
  
  
    There is an emerging need to know how a given model was pre-trained:  fp16, fp32, bf16. So one won’t try to use fp32-pretrained model in fp16 regime. And most recently we are bombarded with users attempting to use bf16-pretrained (bfloat16!) models under fp16, which is very problematic since fp16 and bf16 numerical ranges don’t overlap too well. 
We are discussing adding a new field to  models that will tell users how it was trained. 
Some papers don’t disclose how a model was trained. So …