Solved by seungjun in post #2 
              
              
                I’m not sure if this is the reason but PyTorch recommends putting backward computation out of the autocast context. 
Here’s a quote from the pytorch documentation. 
autocast should wrap only the forward pass(es) of your network, including the loss computation(s). Backward passes under autocast are n…