anyone notice large difference of using fp16 ?
I recently noticed that using fp16 dropped the performance of BERT on my own dataset but improved on another (it works fine on examples like MPRC). It's about 4% so unlikely to be random noise.

I'm trying to see the reason and noticed examples from apex:
https://github.com/NVIDIA/apex/tree/master/examples
actually uses a global copy of the parameters during training but examples in this repository just use fp16 for all steps (and for saving parameters.) Wondering will this be the potential reason?