It’s been a long time since this post, but maybe you remember if the problem with fp16 will appear when training the models from scratch (pretraining)?
I’ve seen some NaNs already while training with fp16 on, but after lowering the learning rate, beginning of training looks reasonable.