Thanks for replying!! I am using apex O1 opt level, and yes I tried the new pytorch 1.6 feature on the same platform as the original apex codebase. The pytorch native fp16 training is slower than the apex based fp16 training.