Solved by seungjun in post #2 
              
              
                PyTorch has a list of operations that can autocast to fp16. The operations not listed here will remain in fp32. Batch normalization will stay in fp32 when you use amp.autocast(). 
PyTorch native amp is similar to apex level O1. A more detailed explanation of @mcarilli can be found here.