Upgrade to new FP16
The original FP16_Optimizer and the old “Amp” API are deprecated and subject to removal at any time. Should we consider moving to the new one?

https://nvidia.github.io/apex/amp.html#for-users-of-the-old-fp16-optimizer
Just saw run_glue has the new one.