There is an emerging need to know how a given model was pre-trained:  fp16, fp32, bf16. So one won’t try to use fp32-pretrained model in fp16 regime. And most recently we are bombarded with users attempting to use bf16-pretrained (bfloat16!) models under fp16, which is very problematic since fp16 and bf16 numerical ranges don’t overlap too well 18.
We are discussing adding a new field to  models 21 that will tell users how it was trained.
Some papers don’t disclose how a model was trained. So perhaps other ways can be found.
I made this into a wiki post, so if you can help to compile the knowledge base - and you know for sure which mode it belongs too and you can cite a reference - please add an entry below.
Notes:

Let’s focus on official models here - i.e. ones with papers (since some models have hundreds of derived/finetuned checkpoints)
One entry per model is enough unless you know that different checkpoints were trained with different precision which is for example the case with EleutherAI/gpt-neo checkpoints (one in bf16 and another in fp32)
Typically if a model was trained on TPU v2 or higher it’s almost sure it’s bfloat16.
We are looking for definitive data with references where it clearly states how the model was trained. If you are not sure then please don’t add anything.

Thank you!
This is a WIKI post so please add the data directly

Precision of Pre-trained Models (Wiki)

float16 (mixed precision)


allenai/longformer - paper 4, “we employed mixed precision training (floating points 16 and 32) using apex12 to reduce memory consumption and speed-up training. However, we kept the attention computation in fp32 to avoid numerical instability issues.”

allenai/led - same as allenai/longformer


lvwerra/codeparrot - informed by the creator of the model



float32


EleutherAI/gpt-neo-2.7B - stated by creators in this issue 16.

gsarti/it5-base and other it5-* - stated by creator (JAX-trained)


bfloat16


google/mobilebert - paper 2, “we train IB-BERTLARGE on 256 TPU v3 chips”

EleutherAI/gpt-neo-1.3B - stated by creators in this issue 16. (XXX: this looks wrong)

google/pegasus-cnn_dailymail - XXX: needs reference

google/pegasus-xsum - XXX: needs reference

google/mt5 - most likely same as t5

t5 -  paper 3 “TPU v3 chips”

bigscience/T0 and other T0* models (trained on TPUs, confirmed on bigscience slack)